Sara Colombo, Yihyun Lim, Miguel Bruns Alonso, Lin-Lin Chen, Tom Djajadiningrat, Loe Feijs, Jun Hu, Steven Kyffin, 
Elif Ozcan, Lucia Rampino, Edgar Rodriguez Ramirez, Dagmar Steffen 


Design and Semantics 
of Form and Movement 


DeSForM 2019 
Beyond Intelligence 


| am Massachusetts [Ty] Northumbria [rere 
L| Institute of Eu University HOCHSCHULE 
NEWCASTLE LUZERN 


Technology 


# WIN TE WHARE WANANGA O TE ÜPOKO OTE IKA A MAUI "rn 
IM ^ AgBUCIORA —TUDelft 


DIPARTIMENTO DI DESIGN 


Cover image: Aposema (Adi Meyer, Sirou Peng, Silvia Rueda, "Aposema: Exploring Communication in an Apathetic 
Future", page 241) 


DeSForM 2019 is supported by: 


9 
rw Comparative Media Studies | Writing 


EBEN M h 
| l | T pone of MIT 
Technology Desi gn 


Lab 


we 


TE WHARE WANANGA O TE ÜPOKO O TE IKA A MAUI vie giy Northumbria 
Epi VICTORIA TU/e University 


NEWCASTLE 


Lucerne University of 
Applied Sciences and Arts 


OO 
POLITECNICO 
HOCHSCHULE MILANO 1863 TUDelft 
LU Z E R N DIPARTIMENTO DI DESIGN 


With the financial support of Google AI, Philips, Research Center on Artificial Intelligence in Value 


Creation - Emlyon Business School, and Northeastern University Center for Design. 


Google Al PHILIPS fron 


business 


AIM 
school institute 


Northeastern University 
Center for Design 


ISBN 978-0-578-60379-7 
ISSN 2706-6150 


Design and Semantics 
of Form and Movement 


DeSForM 2019 
Beyond Intelligence 


Edited by 

Sara Colombo 
Yihyun Lim 

Miguel Bruns Alonso 
Lin-Lin Chen 

Tom Djajadiningrat 
Loe Feijs 

Jun Hu 

Steven Kyffin 

Elif Ozcan 

Lucia Rampino 
Edgar Rodriguez Ramirez 


Dagmar Steffen 


DeSForM 2019 Program Committee 


General Conference Chairs 


Sara Colombo, 
Massachusetts Institute of 
Technology, USA 
Yihyun Lim, 
Massachusetts Institute of 
Technology, USA 


Panel Chair 

Beyond Digital: 
Designing with Living 
Things 


Scott Penman, 
Massachusetts Institute of 
Technology, USA 


DeSForM Steering 
Committee 


Miguel Bruns Alonso, 
TU Eindhoven, The 
Netherlands 

Lin-Lin Chen, 

National Taiwan University 
of Science and Technology, 
Taiwan; TU Eindhoven, The 
Netherlands 

Sara Colombo, 
Massachusetts Institute of 
Technology, USA 

Tom Djajadiningrat, 
Philips Design Eindhoven, 
The Netherlands 

Loe Feijs, 

TU Eindhoven, The 
Netherlands 

Jun Hu, 

TU Eindhoven, The 
Netherlands 

Steven Kyffin, 
Northumbria University UK 
Yihyun Lim, 
Massachusetts Institute of 
Technology, USA 

Elif Ozcan, 

TU Delft, The Netherlands 
Lucia Rampino, 


Politecnico di Milano, Italy 


Edgar Rodriguez 
Ramirez, 

Victoria University of 
Wellington, New Zealand 
Dagmar Steffen, 
Lucerne University of 
Applied Sciences and Arts, 


Switzerland 


DeSForM 2019 Program 


Committee 


Galo Canizares, 

Ohio State University, USA 
Jonathan Crisman, 
University of California Los 
Angeles, USA 

Dina EI-Zanfaly, 
Massachusetts Institute of 
Technology, USA 
Antonio Furgiuele, 
Wentworth Institute of 
Technology, USA 
Markus Funk, 
Technische Universitat 
Darmstadt, Germany 
Grace Jun, 

Parsons School of Design, 
USA 

Cindy Hsin-Liu Kao, 
Cornell University, USA 
Sotirios Kotsopoulos, 
Massachusetts Institute of 
Technology, USA 
Eduardo Pellanda, 
Pontificia Universidade 
Católica do Rio Grande do 
Sul, Brazil 

Thijs Roumen 

Hasso Plattner Institute, 
Germany 

Harpreet Sareen, 
Parsons School of Design, 
USA 

Davide Spallazzo, 
Politecnico di Milano, Italy 
Secil Ugur Yavuz, 

Free University of Bozen, 


Italy 


2 Design and Semantics of Form and Movement 


Paolo Volontè, 


Politecnico di Milano, Italy 


DeSForM 2019 Local 


Committee 


Lizzie Bundy, 
William McCoy, 
Administrative Assistants, 
Massachusetts Institute of 
Technology 

Paolo Boni, 

Visiting Graduate Student, 
Politecnico di Milano / 
Massachusetts Institute of 


Technology 


Program DeSForM 2019 


15.00 


15.30 


15.45 


16.30 


16.45 


18.00 


8.15 


9.00 


9.10 


9.30 


10.30 


10.45 


11.45 


October 9, 2019 
Venue: Samberg Center, 6th Floor, E52 MIT Campus. 50 Memorial Drive, Cambridge MA 02139. 


Conference Registration 


Welcome & Introduction 


* Yihyun Lim, Sara Colombo - Conference Chairs, MIT Design Lab 

Panel Presentations: "Beyond Digital: Designing with Living Things" 

* Orkan Telhan, Cofounder, Chief Design & Technology Officer Biorealize Inc. 
* Jiwon Woo, Biodesigner, Hypha Design 

* Jorge Duro-Royo, Co-Director, DumoLab 


Break 


Panel Discussion / Q&A 
* Moderated by Scott Penman, MIT Design Lab 


Conference Reception 


October 10, 2019 
Venue: Samberg Center, 6th Floor, E52 MIT Campus. 50 Memorial Drive, Cambridge MA 02139. 


Conference Registration & Breakfast 


Welcome 


* Federico Casalegno, MIT Design Lab 


Conference Opening Words 
* Sara Colombo, Yihyun Lim - Conference Chairs, MIT Design Lab 


Keynote Presentation "Al as Tool, Partner, and Inspiration" 
* Martin Wattenberg, Google PAIR 


Coffee Break 


Paper session | - Design Manifestos: What's Next | Chairs: Sara Colombo and Yihyun Lim 


* Future Forecasting Wicked Problems: A New Framework for Design 

Fillippo Sanzeni, Ashley Hall, Paul Anderson (Royal College of Art, London) 

* Eventual Design for an Emergent World 

athan Felde (Northeastern University, Boston) 

* The Decentralization Turns in Design: An Exploration Through the Maker Movement 


assimo Menichinelli (RMIT University, Barcelona; Aalto University, Helsinki), 


Priscilla Ferronato (University of Illinois, Urbana-Champaign) 


Coffee break 


Design and Semantics of Form and Movement 


Program DeSForM 2019 


12.00 


12.40 


14.30 


14.40 


Paper session 2 - Interacting with Domestic Intelligences | Chair: Edgar Rodriguez Ramirez 


* The Domestic Shape of Al: A Reflection on Virtual Assistants 
Davide Spallazzo, Martina Scianname, Mauro Ceconello (Politecnico di Milano) 
* Conversational Smart Products: a Research Opportunity, First Investigation and Definition 


llaria Vitali, Venanzio Arquilla (Politecnico di Milano) 


Lunch * Short Paper / Demo Session 


* Prosumeristic Publications: alttyd 

Harshali Paralikar, Ajitesh Lokhande (National Institute of Design, Paldi, Ahmedabad, Gujarat, India), 

* Swimming Coach: An Immersive Swimming Learning System 

Shuo Li, Cheng Yao, Mingxuan He, Qingcong Wang, Ying Wang, Yuyu Lin, Juanli Liu (Zhejiang University, 
Hangzhou), Fan Xia (Mercyhurst Preparatory School, Erie), Leijing Zhou (Zhejiang University, Hangzhou) 

* Designing Transparent Collaborations - Weave 

Gissoo Doroudian (College for Creative Studies, Detroit) 

* Huxley: Intelligent Book as Essentialist Artefact 

David Ramsay, Joe Paradiso (Massachusetts Institute of Technology, Cambridge) 

* OlfacEnhancer: A Vision-Based Scented Necklace for Cross-Modal Perception and Olfaction Augmentation. 
Yuyu Lin, Kai Zheng, Lijuan Liu, Yang Chen, Jiahao Guo, Shuo Li, Cheng Yao (Zhejiang University, Hangzhou), 
Fangtian Ying (Hubei University of Technology, Wuhan) 

* APOSEMA: Exploring Communication in an Apathetic Future 

Adi Meyer, Sirou Peng, Silvia Rueda (University College London) 

* HuValue: A Toolkit to Facilitate Considering Various Human Values in a Design Process 

Shadi Kheirandish, Mathias Funk, Stephan Wensveen (Eindhoven University of Technology), 

Maarten Verkerk (Maastricht University), Matthias Rauterberg (Eindhoven University of Technology) 

* Playing with Systems: Tactile Games as System Prototypes 

Tom Maiorana (University of California, Davis) 

* Attributes of Aliveness: A Case Study of Two Interactive Public Art Installations 

Humbi Song, Oliver Luo, Allen Sayegh (Harvard University, Cambridge) 

* Understanding User Customization Needs: Requirements for an Augmented Reality 

Lamp Customization Tool 

Ana Carina Palumbo, Hella Kriening, Barbara Wajda (Eindhoven University of Technology), 

Monica Perusquía-Hernández (Eindhoven University of Technology; NTT Communication Science Laboratories) 
* Speculating on the Future of Graphic Design in the Age of Intelligent Machines 

Sekyeong Kwon, Robyn Cook (Falmouth University) 

* Al-Stylist: An Al-based Framework for Clothing Aesthetic Understanding 

Xingxing Zou, Waikeung Wong (The Hong Kong Polytechnic University) 


Introduction of AIM Institute Research Initiatives: Artificial Intelligence in Value Creation 


Margherita Pagani, Research Center on Artificial intelligence in Value Creation - AIM Institute - Emlyon Business School 


Paper Session 3A - Interacting with Urban Intelligences (I) | Chairs: Yihyun Lim and Sara Colombo 


* Al-to-Microbe Architecture: Simulation, Intelligence, Consciousness 

Dennis Dollens (Universitat Internacional de Catalunya, Barcelona) 

* Envisioning and Questioning Near Future Urban Robotics 

Maria Luce Lupetti (Delft University of Technology), Nazli Cila (Amsterdam University of Applied Sciences) 

* Robot Citizenship: a Design Perspective 

Maria Luce Lupetti, Roy Bendor (Delft University of Technology), Elisa Giaccardi (Umea Institute of Design) 


4 Design and Semantics of Form and Movement 


Program DeSForM 2019 


15.30 Coffee break 


15.45 Paper Session 3B - Interacting with Urban Intelligences (Il) | Chair: Scott Penman 


* Towards Transparency Between the Autonomous Vehicle and the Pedestrian 
Selin Zileli, Stephen Boyd Davis, Jiayu Wu (Royal College of Art; Intelligent Mobility Design Centre, London) 
* The Coerced User and the Era of Smart City Dissonance 


Guy Cherni, Roee Bigger (Bezalel Academy of Arts and Design, Jerusalem) 
16.30 Keynote Presentation "How to Design for the Unconscious" 
* Matthias Rauterberg, Full professor for "Interactive Systems Design", Department of Industrial Design , 
Eindhoven University of Technology 
18.00 Conference Dinner - Charles River Sunset Cruise Dinner 
October ll, 2019 
Venue: Bartos Theater, Building EI5 Lower Level, 20 Ames Street, Cambridge MA 02139. 


8.15 Conference Registration & Breakfast 


9.00 Keynote Presentation: "Adaptive Dynamics: Creating Intelligent Sportswear Experiences " 
* Charles Johnson, Global Director Innovation, PUMA 


10.00 Paper Session 4 - New Interfaces for Complex Ecosystems | Chair: Davide Spallazzo 


* Drawing Interfaces. When Interaction Becomes Situated and Variable 

llaria Mariani (Politecnico di Milano), Tommaso Livio (Thingk), Umberto Tolino (Politecnico di Milano, Thingk) 
* Individual Mid-Air Gesture Sets Informed by Conceptual Metaphors: A Case Study on How Users 
Generate Mid-Air Gesture Sets to Control Video Streaming 

Gulben Sanli Eren (Istanbul Technical University) 

* A Pedagogy for Noticing — Soma Literacy and the Designer 

Stephen Neely (Carnegie Mellon University, Pittsburgh) 


11.00 Coffee break 


11:20 Introduction of Northeastern University Center for Design 


Paolo Ciuccarelli, College of Art, Media and Design - Northeastern University 


11.30 Paper Session 5 - Smart and Multi-Sensory Systems for Behavior Change | Chair: Lucia Rampino 


* Designing Phygital Activities in a Smart Multisensorial Room: A Collaborative Cognitive Environment 
for Children with and without Disabilities 

Micol Spitale, Agnese Piselli, Franca Garzotto, Barbara Del Curto (Politecnico di Milano) 

* Recommendations when Designing to Address Procrastination: A Psychological Perspective 

Helen Andreae (Northumbria University, Newcastle upon Tyne; Victoria University of Wellington), 

Abigail Durrant, Steven Kyffin (Northumbria University, Newcastle upon Tyne) 

* R2S: Designing a Public Augmented Printed Media System to Promote Care Home 

Residents' Social Interaction 


Kai Kang, Jun Hu, Bart Hengeveld, Caroline Hummels (Eindhoven University of Technology) 


Design and Semantics of Form and Movement 


Program DeSForM 2019 


12.30 


13.15 


14.15 


15.35 


15.50 


16.50 


6 


Lunch 


Participatory Workshop: "The Soma Literacy of AI" 


* Stephen Neely (Carnegie Mellon University, Pittsburgh) 


Paper Session 6 - Design and Semantics for Health and Inclusion | Chair: Sotirios Kotsopoulos 


* Nova Creatio: A Clinical Perspective on Rehabilitative Everyday Objects for People with Chronic Stroke 
Mailin Lemke, Edgar Rodríguez Ramírez, Brian Robinson (Victoria University of Wellington) 

* The Semantics of Conspicuity: Design Strategies to Address Conspicuity in Type | Diabetes Medical 
Devices for Adolescents 

Madeleine J. Hazelton, Gillian M. McCarthy, Edgar R. Rodríguez Ramírez (Victoria University of Wellington) 

* Sitting Still: Seat Design for a New Head-Only MRI Scanner 

Christy Wells, Edgar Rodríguez Ramírez, Mailin Lemke, Benjamin Parkinson (Victoria University of Wellington) 
* Designing Research Prototype for the Elderly: A Case Study 

Cun Li, Jun Hu, Bart Hengeveld, Caroline Hummels (Eindhoven University of Technology) 


Coffee break 


Paper Session 7 - Designing with Humans, Machine Intelligence, and Data | Chair: Scott Penman 


* Plug-ins Jungle: Algorithmic Design as Inbuilt Dynamism Between Human and Artificial Creativity 
Giuseppe Bono (University College London), Pilar Maria Guerrieri (Politecnico di Milano) 
* Defining a Data Impact Tool for Design Courses 


Laura Varisco, Margherita Pillan (Politecnico di Milano), Patrizia Marti (Università' degli Studi di Siena) 


Conference Closing Ceremony 


October 12, 2019 


Boston Brewing & Beyond - Craft Beer Brewery Tour of Boston 


Design and Semantics of Form and Movement 


Contents 


010 


017 


018 


019 


020 


022 


026 


035 


043 


052 


060 


070 


078 


Opening Words 


"Beyond Intelligence: Re-focusing on Human Experience in Complex Artificial Ecosystems" 


Sara Colombo, Yihyun Lim 


Keynotes 


“Al as Tool, Partner, and Inspiration" 
Martin Wattenberg 


"How to Design for the Unconscious" 
Matthias Rauterberg 


"Adaptive Dynamics: Creating Intelligent Sportswear Experiences " 


Charles Johnson 


Panel 


"Beyond Digital: Designing with Living Things" 


Orkan Telhan, Jiwon Woo, Jorge Duro-Royo 


Workshop 


"The Soma Literacy of AI" 
Stephen Neely 


Topic 1 - Design Manifestos: What's Next 


"Future Forecasting Wicked Problems: A New Framework for Design" 


Filippo Sanzeni, Ashley Hall, Paul Anderson 


"Eventual Design for an Emergent World" 
Nathan Felde 


"The Decentralization Turns in Design: An Exploration Through the Maker Movement" 


Massimo Menichinelli, Priscilla Ferronato 


Topic 2 - Interacting with Domestic Intelligences 


"The Domestic Shape of Al: A Reflection on Virtual Assistants" 


Davide Spallazzo, Martina Scianname, Mauro Ceconello 


"Conversational Smart Products: a Research Opportunity, First Investigation and Definition" 


llaria Vitali, Venanzio Arquilla 


Topic 3 - Interacting with Urban Intelligences 


"Al-to-Microbe Architecture: Simulation, Intelligence, Consciousness" 


Dennis Dollens 


"Envisioning and Questioning Near Future Urban Robotics" 


Maria Luce Lupetti, Nazli Cila 


Design and Semantics of Form and Movement 


Contents 


087 


096 


104 


114 


122 


132 


140 


149 


159 


168 


177 


186 


195 


204 


"Robot Citizenship: a Design Perspective" 
Maria Luce Lupetti, Roy Bendor, Elisa Giaccardi 


"Towards Transparency Between the Autonomous Vehicle and the Pedestrian" 
Selin Zileli, Stephen Boyd Davis, Jiayu Wu 


"The Coerced User and the Era of Smart City Dissonance" 


Guy Cherni, Roee Bigger 


Topic 4 - New Interfaces for Complex Ecosystems 


"Drawing Interfaces. When Interaction Becomes Situated and Variable" 


llaria Mariani, Tommaso Livio, Umberto Tolino 


"Individual Mid-Air Gesture Sets Informed by Conceptual Metaphors: A Case Study on How Users 
Generate Mid-Air Gesture Sets to Control Video Streaming" 


Gulben Sanli Eren 


"A Pedagogy for Noticing — Soma Literacy and the Designer" 
Stephen Neely 


Topic 5 - Smart and Multi-Sensory Systems for Behavior Change 


"Designing Phygital Activities in a Smart Multisensorial Room: A Collaborative Cognitive Environment 
for Children with and without Disabilities" 


Micol Spitale, Agnese Piselli, Franca Garzotto, Barbara Del Curto 


"Recommendations when Designing to Address Procrastination: A Psychological Perspective" 


Helen Andreae, Abigail Durrant, Steven Kyffin 


"R2S: Designing a Public Augmented Printed Media System to Promote Care Home Residents’ Social 
Interaction" 


Kai Kang, Jun Hu, Bart Hengeveld, Caroline Hummels 


Topic 6 - Design and Semantics for Health and Inclusion 


"Nova Creatio: A Clinical Perspective on Rehabilitative Everyday Objects for People with Chronic 
Stroke" 


Mailin Lemke, Edgar Rodriguez Ramirez , Brian Robinson 


"The Semantics of Conspicuity: Design Strategies to Address Conspicuity in Type | Diabetes Medical 
Devices for Adolescents" 


Madeleine J. Hazelton, Gillian M. McCarthy, Edgar R. Rodriguez Ramírez 
"Sitting Still: Seat Design for a New Head-Only MRI Scanner" 


Christy Wells, Edgar Rodriguez Ramirez, Mailin Lemke, Benjamin Parkinson 


"Designing Research Prototype for the Elderly: A Case Study" 


Cun Li, Jun Hu, Bart Hengeveld, Caroline Hummels 


Topic 7 - Designing with Humans, Machine Intelligence, and Data 


"Plug-ins Jungle: Algorithmic Design as Inbuilt Dynamism Between Human and Artificial Creativity" 


Giuseppe Bono, Pilar Maria Guerrieri 


Design and Semantics of Form and Movement 


Contents 


211 "Defining a Data Impact Tool for Design Courses 


Laura Varisco, Margherita Pillan, Patrizia Marti 


Interactive Demos 


220 “Prosumeristic Publications: alttyd” 


Harshali Paralikar, Ajitesh Lokhande 


224 "Swimming Coach: An Immersive Swimming Learning System" 


Shuo Li, Cheng Yao, Mingxuan He, QingcongWang, Ying Wang, Yuyu Lin, Juanli Liu, Fan Xia, Leijing 
Zhou 


228 "Designing Transparent Collaborations - Weave" 


Gissoo Doroudian 


233 "Huxley: Intelligent Book as Essentialist Artefact" 


David Ramsay, Joe Paradiso 


237 "OlfacEnhancer: A Vision-Based Scented Necklace for Cross-Modal Perception and Olfaction 
Augmentation" 


Yuyu Lin, Kai Zheng, Lijuan Liu, Yang Chen, Jiahao Guo, Shuo Li, Cheng Yao, Fangtian Ying 


241 "APOSEMA: Exploring Communication in an Apathetic Future" 
Adi Meyer, Sirou Peng, Silvia Rueda 


246 X "HuValue: A Toolkit to Facilitate Considering Various Human Values in a Design Process" 


Shadi Kheirandish, Mathias Funk, Stephan Wensveen, Maarten Verkerk, Matthias Rauterberg 


250 "Playing with Systems: Tactile Games as System Prototypes" 


Tom Maiorana 


254 "Attributes of Aliveness: A Case Study of Two Interactive Public Art Installations" 
Humbi Song, Oliver Luo, Allen Sayegh 


258 "Understanding User Customization Needs: Requirements for an Augmented Reality Lamp 
Customization Tool" 


Ana Carina Palumbo, Hella Kriening, Barbara Wajda, Monica Perusquía- Hernández 


262  "Speculating on the Future of Graphic Design in the Age of Intelligent Machines" 


Sekyeong Kwon, Robyn Cook 


266  "Al-Stylist: An Al-based Framework for Clothing Aesthetic Understanding" 
Xingxing Zou, Waikeung Wong 


Design and Semantics of Form and Movement 


Opening Words 


Sara Colombo, Yihyun Lim 


Design Lab, Massachusetts Institute of Technology, USA 


Beyond Intelligence 


Re-focusing on Human Experience in Complex Artificial Ecosystems 


Designing Ecosystemic User Experiences 


In recent decades, design has faced profound challenges and transformations. The traditional approach 
to crafting and shaping the tangible world has been challenged by the world's infusion with digital 
technologies, which have made it smarter, more interactive, and more connected. 

DeSForM, the conference on the Design and Semantics of Form and Movement, was undertaken in 
2005 as an attempt to foster discussion in the design community around how to design the meaning, 
aesthetics, and experience of responsive and dynamic artifacts. DeSForM's intent was to 

"present current research into the nature, character and behaviour of emerging new typologies of co-designed, 


content rich, connected and intelligent objects within adaptive systems." ' 


Those 'emerging new typologies' of 'intelligent objects' have developed and spread over the following 
years, bringing the rise and formalization of new areas of design research, such as interaction design, 
user experience, and the aesthetics of interaction. These domains have been widely investigated in the 


works presented and debated in the past editions of DeSForM. 


However, recent technological developments are causing even more rapid and extreme changes than 
the ones witnessed at the beginning of this century. The emergence of artificial intelligence and machine 
learning, flexible electronics, virtual and augmented reality, miniaturized and implantable sensors, and 
hybrid synthetic-biological materials have not only provided designers with new design ingredients, but 


also generated new cultural and social landscapes in which they must operate. 


1 https//www.northumbria.ac.uk/about-us/academic-departments/northumbria-school-of-design/research/desform/ 
what-is-desform/ 

? For a full account of the DeSForM past editions, you can access the previous proceedings at https://www.northumbria. 
ac.uk/about-us/academic-departments/northumbria-school-of-design/research/desform/previous-conferences-and- 


proceedings/ 
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In this context, designers are called to design not intelligent products within adaptive systems, 

but rather those adaptive systems as a whole. Objects can no longer be interpreted and designed 

as independent elements, detached from the other components of the complex digital-physical 
ecosystems they belong to. In such hybrid ecosystems, new distributed intelligences, advanced materials 
and interfaces, sensing technologies, data, and humans are deeply interconnected and mutually 

shaped. Their understanding, design, and evaluation demand approaches and tools able to tackle this 
complexity. Despite this, as these systems become increasingly intelligent, their meanings, aesthetics, 


and ethics still seem to be overlooked. 


Designing beyond intelligence means that the design of such complex and smart ecosystems should 
consider issues beyond mere algorithmic thinking and functionality. Scholars and practitioners in 
the design field are encouraged to reflect on the connections and mutual relations between the 
performance of these intelligent ecosystems and their physical appearances, meanings, personalities, 
and interaction modalities. In doing so, they will be able to address the design of ecosystemic user 


experiences. 


Going ‘Beyond Intelligence’: New Challenges for Design 


In the XI edition of DeSForM, hosted by the Massachusetts Institute of Technology, we explore the 
implications of recent and emerging technological transformations in the practice of design, with a 


particular focus on the human experience of these complex systems. 


We invited designers, artists, researchers, and industry practitioners to address the need to design for 
distributed, hyperconnected, and learning intelligent ecosystems, and to investigate how their meanings, 


experience, and ethics can be approached. 


In doing so, we identified a number of possible challenges that we believe are worth exploring in the 
upcoming years. They refer to i) the growing complexity of the concept of user experience; ii) emerging 
forms of interaction with human-like intelligences; iii) the ethical implications of digital-physical systems; 


and iv) the new roles designers should assume in this context. 
Experiencing Complexity 


As ecosystems of digital-physical solutions become more layered, distributed, and connected, the user 
experience also grows in complexity. New elements need to be considered, including the meanings of 
these systems, the multisensory and multimodal interactions they necessitate, and the emotions that 
such interactions generate. 

The tangible manifestations of the systems users interact with are just a tiny part of a huge underlying 
infrastructure of data, algorithms, platforms, and digital contents. As functions overlap in the same 
product and digital contents constantly change, physical objects become just the medium for a 
plethora of meanings derived from multiple connected platforms. What is the role of aesthetics in such 
dynamic, digital-physical ecosystems? What meanings can tangible forms convey? What new tools and 


frameworks are needed to design and evaluate the growing complexity of user experience? 
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Interacting with New Intelligences 


Artificial intelligence opens up new frontiers for design, where emerging forms of distributed 
intelligence become design material. Technological advancements in this field now provide the 

user with the possibility to have increasingly human interactions with non-human artifacts. Users' 
interactions with specialized intelligences have progressively taken on the appearance of companionship 
and assistantship, as seen in the rise of chatbots and social robots. How does artificial intelligence 
transform artifacts (objects, spaces) and their interaction modalities? How can design give meaning 

and form to artificial intelligence, when embedded into products? How do artifacts' aesthetics and 
experience change through Al at home, at work, or in public spaces? This material should be fully 
investigated in terms of tangible manifestations, social implications, and impact on the design process 


and the user experience. 


Societal Impacts 


The above-mentioned transformations surely pose new challenges to design, not only in shaping 

the tangible forms of these systems, their meanings, and aesthetics, but also in anticipating the 
consequences they might have on humans at the individual and societal levels. The emergence of 

Al, robotic solutions, and big data connected with the spaces, objects, and people we interact with 
everyday will create new landscapes for future generations of designers. This will require designers 
to adopt new lenses in the design and evaluation of emerging technology, and it will necessitate that 
designers equip themselves with new ethical paradigms. 

How will algorithmic decision making and autonomous systems impact user experience and behavior? 
How can we design for transparency and reliability? What are the long-term effects of new digital 
technologies on society? 

As systems become smarter, more self-governed, and increasingly embedded into our reality, designers 


should develop new approaches and methods to consider ethical issues in their practice. 


Future Roles of Design 


This evolving context calls for new design skills and ways of thinking that go beyond the traditional 
field of design. How will this domain change, in order to interface with new fields of knowledge such 
as biotechnology, computer science, Al, and ethics? What are the future roles of design in shaping the 
growing complexity of the artificial world, where the boundaries between artificial, human, and natural 
fade? What role can designers play in the multi-disciplinary teams that will envision future systems 


which are more and more interactive, interconnected, and even unpredictable? 


While some of these challenges are just emerging, other issues seem to be already compelling, or will 
likely be in a short time. This conference invites the design community to reflect deeply on the current 
and future transformations enabled by technology, as well as their effects on design itself, and on 
society as a whole. It will take time to fully understand this new landscape and its effects on humans, 
and to approach it with a critical eye. As some of the works included in these proceedings point out, 
design is just now starting to react to this transitional moment and to equip itself with new sets of 


concepts, approaches, and methods to face this changing reality. 
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DeSForMI9 Proceedings 


The proceedings from the DeSForM | Beyond Intelligence conference are structured to address some of 


the emerging questions raised above. 


Design Manifestos. The first section of the proceedings is dedicated to sharing thoughts on the roles 
of designers and the meanings of design practice through a series of design manifestos. What are new 
tools, methods, and frameworks that allow designers to forecast and solve the wicked problems of the 
future? Observing the current landscape of complex systems and varied forms of intelligences - from 
artificial machines to synthetic biology - some 'turns' in design practice are identified, which led us 

to the current state. These manifestos question the role of the designer and the meaning of agency 

as design practice becomes collaborative at all stages, especially through the use of algorithmically 
enhanced design tools and artificial intelligence. These questions are expected to repeatedly rise to the 


forefront throughout the conference. 


Interacting with Domestic Intelligences. Perhaps we are more accustomed to interacting with 
artificial intelligence than we think. Virtual assistants and conversational agents are slowly becoming 
the norm in domestic settings through the use of smart connected products and social robots. The 
works presented in this section analyze the current product landscape of domestic intelligences and 
provide an initial understanding of relationships between form, function, and meaning. As we move into 
the future of embedded intelligences in our everyday environments, how should we design the shape 
and interaction modalities of artificial intelligence to effectively translate its function and meaning in an 


intuitive way? 


Interacting with Urban Intelligences. From autonomous vehicles to delivery robots, we will soon, 
if we are not already, be sharing our urban environment with other intelligent entities. We can no 
longer opt out from this smart environment experience, and we have no other option but to interact 
with such systems and provide resources (data) back into their digital networks. Papers collected in 
this section explore the opportunities and affordances that become available in the design of such 
environments, as we learn to coexist with various forms of artificial intelligence. From the concept of 
'robot citizenship' to that of 'coerced' users, authors suggest new approaches to bring the perspective 
of urban robots, citizens, and other autonomous systems into the design process. 

Shifting our view to the building scale, various forms of intelligence will also become embedded in our 
architecture. The rise of synthetic biology and the use of engineered microbes as building blocks in 
urban architecture is opening up an era of hybrid buildings, which function as metabolic systems. Can 


we simulate nature and create environmentally performative, intelligent, and living buildings? 


New Interfaces for Complex Ecosystems. In this plethora of complex ecosystemic experiences, 
what are the new interfaces for control and interaction? Expanding from voice, text, and gesture-based 
modalities, what novel interactions can we design for? As we start to build a dictionary of universal 
interactions with smart products, how shall we explore the semantics of interaction language? Here 

we explore research in the design of interaction metaphors to represent conceptual understanding of 
situations (and translation of our language) to communicate intuitively with smart devices. But perhaps 
before we attempt to design new languages, we should think about methods to reveal and recognize the 


affordances of technologies in relation to our body, and the aesthetics of interactions it brings. 
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Smart and Multi-Sensory Systems for Behavioral Change. Although the experiencing human 
body is the constant in this ever-changing environment of complex ecosystems of intelligences, the 
designed experience of these systems induces behavioral changes in its users. The papers in this section 
investigate the effects of the merger of digital experiences and physical environment on human behavior, 
from creating 'phygital' activities that affect the cognitive learning abilities of children, to mitigating 
procrastination through designed interventions in built environment and interactive artifacts. Moreover, 
attention is paid to the experience of caring in elderly living environments through the use of connected 
technologies. What experiences can we augment with technologies and what should remain as human- 


driven? 


Design and Semantics for Health and Inclusion. Focusing on the health industry, we look further 
into the semantics and aesthetics of interaction in medical and assistive devices. This topic explores 
semantic strategies and design criteria to overcome social stigmas in the use of assistive devices, and to 


improve rehabilitation processes as well as overall user experience. 


Designing with Humans, Machine Intelligence, and Data. This final topic brings us back to 

the discussion on the role of design and the various societal issues designers should consider in their 
research and practice. What tools and methods can help us to navigate the complexity of data privacy 
issues in the co-design process? Moreover, as our design software tools become more intelligent and 
generative, should we rethink the notion of design agency, and invite our software tools to become our 


creative partners? 
Interactive Demos 


Short papers and their related interactive demos explore five thematic areas: designing immersive 
experiences, Al and human collaboration, Al curated experiences, sensory augmentation and 


communication, and processes and tools for design and awareness. 


Immersive Experience. Virtual reality is often used in safety training for hazardous situations or 
difficult to access environments. What if mixed-reality experiences are used to learn swimming? Can 
simulated experiences help overcome the fear of water, and bring the experience of swimming to those 
with limited access to aquatic environments? Along the line of making experiences real, another project 
explores the ‘aliveness’ of public art installations, to bring continuous life through embedding real-time 


responsiveness and audience participation in art experience. 


Al and Human Collaboration. Here we explore the design of transparent collaboration between 
humans and machine intelligences. From graphic design to web contents, and also digital publications, 
how do we define design agency when algorithms and machine intelligence become active creators of 


experiences? 


Al Curated Experiences. From music to movie platforms, we are accustomed to algorithmically- 
curated contents based on individual preferences. How about algorithmically-curated clothing 
suggestions? Can machine intelligence evaluate aesthetics, cultural nuances, individual preferences, 
and other design elements? Thinking further about human experiences, can Al understand and foresee 
users' desired engagement level, and curate a holistic reading experience by, for example, selecting 


appropriate content and creating an optimal ambient sensory environment? 
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Sensory Augmentation and Communication. Human experience is deeply affected by sensory 
experiences and social interactions. Can an augmented olfaction device strengthen the link between 
olfaction, vision, and memory? Or better yet, can we translate our emotions and visually communicate 
these through an interactive wearable device for the face? Can digital technology and algorithms 


enhance human communication and social interaction, or is this a false hope? 


Process and Tools for Design and Awareness. We close the Interactive Demo section with new 
toolkits and processes for design and awareness. What are the ways to guide a value-driven design 
process and the creation of meaningful products? What methods can help unpack complex systemic 


challenges like climate change, to stimulate discussion and ideate potential interventions? 


Future Perspectives 


This edition of DeSForM covers a wide range of topics related to designing with new forms of 
intelligence in complex human-artificial and digital-physical ecosystems. If the trends we are debating at 
this venue continue to develop, distributed intelligences could potentially affect any designed reality, as 
well as the experiences that result from users' interactions with these realities. 

While design as a discipline is required to develop specific frameworks and tools to tackle the growing 
complexity of our world, such a diversity of application fields also calls for a collaborative approach 
with other areas of knowledge. Designers will need to operate more and more in concert with 
technologists, computer scientists, architects, social scientists, psychologists, and ethicists, as well 

as policy makers and industry players, in a joint effort to reduce the risks and amplify the positive 
potential of these transformations. This will ultimately support the type of technological development 


that is truly centered on and beneficial to humans, both at the individual and societal levels. 
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Keynotes 


Martin Wattenberg 
Co-lead, Google PAIR (People + Al Research) 


Al as Tool, Partner, and 


Inspiration 


Abstract 

How should people relate to artificial intelligence technology? Is it 
a tool to be used, a partner to be consulted, or perhaps a source 
of inspiration and awe? As technology advances, choosing the right 
human / Al relationship will become an increasingly important 
question for designers. | will show a series of examples--ranging 
from data visualizations to tools for medical practitioners--that 
illustrate how Al can play each of these roles in turn. | will then 
discuss and analyze the considerations that determine which role 


may be right for which situation. 


» 


artin Wattenberg co-leads Google's 
PAIR (People + Al Research) initiative 
and the Big Picture team. His work at 
Google, with long-time collaborator 
Fernanda Viégas, currently focuses on 
making Al technology broadly accessible 


and reflective of human values. He, his 


eam, and Fernanda have also created 


end-user visualizations for products 
such as Search, YouTube, and Google 
Analytics. 

Before joining Google, Fernanda and 
he founded Flowing Media, Inc., a 
visualization studio focused on media 
and consumer-oriented projects. Prior 
o Flowing Media, they led IBM's Visual 
Communication Lab, which created the 
ground-breaking public visualization 
platform Many Eyes. He came to IBM 


from Dow Jones, where he was the 


Director of Research and Development 
at SmartMoney.com. His work there 
included some of the earliest pieces of 
interactive journalism. 
He is known for visualization-based 
artwork, which has been exhibited in 
venues such as the London Institute 


of Contemporary Arts, the Whitney 


useum of American Art, and the New 


York Museum of Modern Art. 


He has a Ph.D. in mathematics from 
U.C. Berkeley, focusing on dynamical 


systems. 
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Keynotes 


G. W. Matthias Rauterberg 
Full Professor for "Interactive Systems Design" 


Department of Industrial Design, Eindhoven University of Technology 


How to Design for the 
Unconscious 


Abstract 

We can distinguish human activities as intentional and unintentional, 
conscious and unconscious, and many more. Most of the modern 
interaction design in the West relies on conscious decision making 
of the user. One challenge in the upcoming design of interactive 
products and systems is how to tap into the unconscious of 
human behavior. Such kinds of design have to rely on input signals 
beyond introspection based on human language. These new types 
of input are bio-signals, video monitoring, deep learning, etc. 

The presentation will address potential building blocks of the 
unconscious and related design challenges. | will show potential 


directions into the future. 
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» 

Prof. Dr Matthias Rauterberg received 
a B.S. in Psychology (1978) at the 
University of Marburg (Germany), 

a B.A. in Philosophy (1981) and a 

B.S. in Computer Science (1983), a 

.S. in Psychology (1981) and a M.S. 
in Computer Science (1986) at the 
University of Hamburg (Germany), 
and a Ph.D. in Computer Science/ 
athematics (1995) at the University 
of Zurich (Switzerland). From 2006 till 
2015 he was the head of the Designed 
ntelligence research group, department 
of Industrial Design at the Eindhoven 
University of Technology (TU/e, 

The Netherlands). He has over 450 
publications in international journals, 
conference proceedings, books, etc. 


is recent research is in the area of 


entertainment computing, cognitive 


systems, human-computer interaction, 
and design science. 


Keynotes 


Charles Johnson 


Global Director Innovation, Puma 


Adaptive Dynamics: 
Creating Intelligent 
Sportswear Experiences 


Abstract 

Adaptive Dynamics: Creating Intelligent Sportswear Experiences 
will discuss PUMA’s legacy in innovation for human performance. 
It will discuss PUMA's journey in integrating the experience of 
Adaptiveness with digital technology, ecosystem of connected 
user-experiences, and new forms of intelligences, such as living 
microorganisms. From track spikes to football kits to automated 
lace footwear and platforms for customized user experiences, this 
presentation will address innovation perspective from the industry 
side, and highlight how PUMA is defining the future vision for 


sportswear in the current landscape of technological developments. 


» 

Charles Johnson has more than 30 
years' experience in the sports industry 
specializing in product strategy and 
innovation. He has served global 
brands such as Adidas, Converse and 
Ralph Lauren both as an employee 
and formerly through Sports Creative 
Group, Inc., the New York-based 
consultancy he founded. He received 
his formal design training at Carnegie 
Mellon. In his current role as PUMA's 
Global Director of Innovation, Charles 
oversees a network of designers, 
engineers, material specialists and 
scientists responsible for bringing life 
to innovative, performance enhancing 


products and systems for athletes. 
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Panel 


Beyond Digital: 
Designing with Living Things 
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Panel 


Orkan Telhan 


» 

Orkan Telhan is an interdisciplinary designer whose investigations focus on the 
design of interrogative objects, interfaces, and media, engaging with critical 
issues in social, cultural, and environmental responsibility. Telhan is an Associate 
Professor of Fine Arts - Emerging Design Practices at University of Pennsylvania, 
Weitzman School of Design. He holds a PhD in Design and Computation from 
MIT's Department of Architecture. He was part of the Sociable Media Group 
at the MIT Media Laboratory and a researcher at the MIT Design Lab. Telhan's 


individual and collaborative work has been exhibited internationally in venues 

including the Istanbul Biennial (2013), Istanbul Design Biennial (2012, 2016), Milano 
Design Week, Vienna Design Week, the Armory Show 2015 Special Projects, Ars 
Electronica (2007, 2017), ISEA, LABoral, Archilab, Matadero Madrid, Architectural 


Association, the Architectural League of New York, MIT Museum, Museum of 


Contemporary Art Detroit, and the New Museum of Contemporary Art, New 
York. 


Jiwon Woo 


» 

Jiwon Woo is a multidisciplinary artist, designer, and researcher based in New York. 
She is a lecturer at the University of Pennsylvania and a founder of Hypha Design, 
based in Korea. Woo investigates the rapidly transforming role of art, design, 

life science, and technology across generations. She researches new biologically 
designed materials and fabrication methods derived from nature and the human 
body. Woo won an Honorary Mention at Ars Electronica Prix 2019 at the 
Artificial Intelligence & Life Art category, and she is also the final winner of Bio Arts 
and Design Award 2017. Woo's work has been exhibited on an international scale 
at the Victoria and Albert Museum in London, London Design Festival 2018, Milan 
Design Week 2018, Ars Electronica Festival 2018, and others. She received MFA 
from University of Pennsylvania, M. Political Science from Yonsei University, and 
B.A., Wellesley College. 


Www.woojiwon.com 


Jorge Duro-Royo 


> 

Jorge Duro-Royo is a PhD at the MIT Media Lab. He works on novel virtual- 
to-physical theoretical and applied digital design methods termed Fabrication 
Information Modeling (FIM). He is an Architect by the Polytechnic University of 
Catalonia, School of Architecture (UPC-ESTAV) and a Mechanical Engineer by the 
Polytechnic University of Catalonia, School of Industrial and Aeronautic Engineering 
(UPC-ETSEIAT) where he graduated with honors with a focus on structural 

design and construction. Since 2010 he has taught introductory hands-on courses 


on Digital Visualization, Parametric Architecture, and Computational Design, and 
collaborated with diverse discipline groups at MIT and Harvard. He co-directs 
DumoLab with Laia Mogas. 
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Workshop 


Stephen Neely 


Assistant Professor in Dalcroze Eurhythmics, Carnegie Mellon 


The Soma Literacy 
of Al 


Stephen Neely, PhD, Carnegie Mellon 
Assistant Professor in Dalcroze 


Eurhythmics, is a teacher, conductor, 
theorist, musician, and clinician who 


teaches, lectures, and presents 


Abstract 


In this short session, Stephen Neely presents methods from theater/ 


workshops in the fields of design, music, 
architecture, and pedagogy. He is a 
music performer training to challenge participants to question the dynamic speaker who enjoys traveling 
ways technologies engage the actual human reality and notice the o present hands-on workshops 
and clinics focusing on the overlaps 


variables of experience we are all pre-reflectively attuned to. : 
between music, design, the body, 
esthetics, performance, and experience. 
He has entertained invitations to 
present his work in Mexico, Indonesia, 
China, England, Switzerland, and 
hroughout the USA. An introduction 
o his research agenda can be found 
hrough his TEDx talk and a more 


horough discussion in his dissertation 


Soma Literate Design—recentering the 
interstitial in experience. 

"My research focuses on the physical 
nature of experience and the 
reflections of the artful gesture in 
everyday interactions—that is, the 
ways in which our feeling bodies 

are necessary components of 

musical participation and how that 
understanding presents artful potential 


in any experience." 
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Topic I. 


Design Manifestos: 
What's Next? 
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Filippo Sanzeni, Ashley Hall, Paul Anderson 
Royal College of Art, London, UK 
filippo.sanzeniQ)network.rca.ac.uk 

ashley.hall@rca.ac.uk 


paul.anderson@rca.ac.uk 


Future Forecasting Wicked Problems: 
A New Framework For Design 


Abstract 


Design can be characterized as ‘knowledge for future 
transformation’ and is a discipline concerned with 
developing new products, systems and services that 
change the future. However, the methods we use for 
generating foresight are underdeveloped and some 
gaps and issues remain. This is especially true between 
more speculative approaches to designing futures and 
applied industrial approaches. This paper explores 
these issues in relation to addressing wicked problems 
in design, specifically the emphasis on qualitative 
methods and how these lack measurable indicators 

of problem improvement. The use of mixed methods 
offers possibilities for combining the power of abductive 
thinking generating alterative visions of the future 
alongside quantifiable improvements. We review 
methods for future forecasting in other fields including 
economics and business management and explore how 
these can be transposed into design practice to address 
some of the issues raised. A proposal is made to achieve 
this via an interdisciplinary mixed method approach by 
instigating a process of gap analysis within a new design 


futures framework. 


Keywords 


Future Forecasting, Wicked Problems, Design Methods 
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1. Introduction 


The argument for future forecasting in design is 

based on the following semantic difference between 
‘prediction’ and ‘forecasting’: predicting refers to 
announcing future events and making prophecies; 
forecasting is about making calculated hypotheses and 
suppositions about what could happen in the future 
using qualitative or quantitative information from the 
past or present. The importance of thinking about 

the future and preparing for it has grown in the last 
decades, due to the increasing awareness and number 
of complex and undefinable problems. These are issues 
which cannot be solved with a miraculous solution, 
simply because the problems are too complex and 
dynamic for solutions to be proposed before we begin 
to tackle the problem. Examples of wicked problems 
include opiate addiction, obesity, climate change, 
anxiety. Further, we recognize that the issues of climate 
change are composed of a complex interacting set of 


wicked problems. 


The type of future forecasting we focus on here is 
specifically aimed towards supporting the improvement 
of wicked problems tackled via a range of design 
methods by designers or those trained in design 
methods from a range of disciplines. These include 
traditional design fields such as industrial design and 
service design but also those working in health, energy, 


social, transport, ICT and other fields. 


Rittel and Webber called these types of issues "wicked 
problems", in contrast to "tame problems" which can be 
approached and solved using known analysis paradigms 
and decision-making processes. A wicked problem is 

a complex and troubling situation, which refuses to 

be fenced and described with a unifying definition and 
requests interdisciplinary solutions [1]. Figure 1 shows a 
diagram from Rittel's ‘The Reasoning of Designers’ and 
is clearly aimed at generating a flow diagram to tackle 
complex problems. However, the type of information 
being processed (qualitative/quantitative) is omitted. 
Buchanan argues that design, is inherently suitable 

for tackling wicked problems since it has a potentially 
universal scope, which is inevitably narrowed down 

into a subject, emerging from the observed issues and 


problems [2]. 


1.1 Designing for the Future 


The field of design is focused on changing the future: 
two early historical examples are Buckminster Fuller's 
1956 course about Comprehensive Anticipatory Design 
Science, which was aimed at systematically foreseeing 
future crisis [3] and Papanek's Design for the Real 
World, which pioneered the field of sustainable design 
by citing the adverse effects on the environment of 


designing products [4]. 


Other established approaches to exploring possible 
futures are discursive design, fictional design and its 
extension - speculative design and critical design (SCD). 
As the name intends, discursive design is aimed at 

the creation of products — intended in the broadest 
meaning possible — which aim to facilitate discourse and 


debate in design practice [5], fictional design creates 


Fig. 1. Rittel's flow diagram from the reasoning of designers (Source: Rittel, H. (1988). The Reasoning of 


Designers, Arbeitspapier zum International Congress on Planning and Design Theory in Boston, August 1987) 
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suspension of disbelief through "profitable", "desirable" 
but not necessarily "buildable" diegetic prototypes [6]. 
Dunne and Raby [7-9] broadened this second method 
by framing the prototypes in society and by stressing 
out their ability to critique the very development 

of the technologies involved. They use extensively 

the concept of Future Cones (Figure 2), initially 
introduced by Hancock and Bezold [10] in 1994 as 
part of a WHO future of health report and later more 
commonly referenced by Voros [11], which projects 
possible, plausible and probable futures and creates an 


opportunity for constructing a preferable future. 


Types of Futures = 


PER 


Short term 
(1-3 years) 


| à Medium term 
(5-20 years) 


Fig. 2. Futures cone (Source: Bezold, C. and Hancock, T. (1994). 
An Overview of the Health Futures Field. WHO Consultation, 
July 19-23. 


Although initially conceived as a critique to the over-use 
of science fiction elements in design, the speculative/ 
critical approach is grounded in the researcher's 
previous experiences, rather than the context in which 
the provocation will take place: two notable examples 
are the project Foragers by Dunne and Raby [12] and 
Burton and Nitta's Republic of Salivation [13]. The first 
was commissioned by the South African organization 
Design Indaba to explore the future of farming and 
represents future Caucasian individuals struggling to 
find food in an over-populated world. As sustainability 
designer Cameron Tonkinwise imagines in one of his 
Science Fictions About Critical Design [14], this project 
might incur some malcontentment if, for example, 

it would have been exhibited in post-ebola Liberia. 

In Republic of Salivation, Burton and Nitta envision a 
future in which food scarcity and hunger is resolved in a 
centralized and government-led rationing system, which 
assign nutrients based on the worker's job. Although it 
resonates with being a possible future dystopia in some 


European countries, this project would simply be an 
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exercise about recent history in other countries, such 


as China, Brazil or even post-war Italy. 


Another approach that looks at designing for the future 
specific to the field of HCI is Reflective Design [15]. 
Based on Critical Theory, this reflective approach 
questions the unconscious assumptions in HCI by 
outlining a series of principles to be considered when 
the designer designs interactions with technology. 
Although these principles can be used to design "better 
answers" [16] for existing and circumscribed problems, 
this method does not lend itself to addressing wicked 


problems. 


This fictional and speculative approach to the future, 
although it may contribute to the creation of stimulating 
and entertaining narratives in some geographical areas 
of the world, is rooted in abductive reasoning [17-19], 
where the designer projects an alternative view of 

the future world hence creating a difference. It can be 
argued that the mere act of projection changes the 
future by offering an alternative that may be adopted. 
As argued above, wicked problems are fluid issues that 
cannot be solved with a preemptive solution. Hence, 
forward-looking yet reductionist design practices, such 
as discursive, fictional and speculative design, are ill- 
equipped when charting a plan of action to tackle them. 
These future generation methods are aimed at inspiring 
debate and consciousness-raising for the consideration 
of the different types of future that we may create. 
However, they do not claim to solve the issues raised. 
Speculative critical design deliberately positions itself 
away from industrial practices and is often exhibited in 
museum contexts in a situation where feedback loops 
for tackling wicked problems are weak. Whilst we can 
see exciting provocative and insightful future projections 
that often visualize future wicked problem scenarios 
these leave a significant gap in addressing wicked 
problems. Improving wicked problems requires the 
integration of a means to assess the changes that have 
taken place either by gathering quantitative data or by 


qualitative assessments throughout the design process. 


Research in the fields of economics and business 
management has employed extensive forecasting 
techniques to explore uncertainty in the future and 
describe the outcomes of decision-making processes 


[20]. The methods employed rely either on qualitative 


insights or quantitative analysis [21]. It has been 
demonstrated empirically that either method if 
employed alone, yields uneven and inconsistent results, 
but the forecast accuracy greatly increases when the 
two methods are combined in a meta-forecast [22-23]. 
In their comprehensive review about the advantages and 
disadvantages of quantitative and qualitative forecasting 
methods, Bunn and Wright concluded that the literature 
on forecasting is overwhelmingly supporting the idea of 
combining multiple outputs, to increase the forecast's 
accuracy, its communicability to third parties and its 


defensibility from criticism [24]. 


Although some examples of using qualitative and 
quantitative methods in interdisciplinary teams exist, 
we found very little focus on over-arching frameworks 
that are specifically tailored for design-led action in 
addressing wicked problems using mixed qualitative 
and quantitative methods throughout the process. We 
propose to transpose the interdisciplinary integration 
of intuitive and computational methods common in 
other fields into the practice of designing for the future 
through a process of gap analysis, to address wicked 


problems. 


1.2 Core Assumptions in Future Forecasting 


In order to assist in the development of a framework 
to integrate qualitative and quantitative methods 
throughout a wicked problem improvement process 
we note here a number of assumptions that function to 


focus our thinking: 


1. The ultimate aim of future forecasting is to improve 
the welfare of humankind, of all animals, plants and 

the biosphere. This is achieved via the systematic 
exploration of possible alternative futures: the probable 
(what is almost sure), the possible (what can be), the 
plausible (what might be) and the preferable (what ought 
to be). However, future forecasting is not merely about 
depicting possible future scenarios: it is the strategic 
approach on how to reach (or avoid) each one of those 


scenarios. 


2. The future of a wicked problem cannot be stated with 
absolute certainty. It is inevitable that a certain degree 
of uncertainty will persist until the forecasted event has 


passed. 


3. Total and organic forecasts are utopic: there will 


always be some aspects of a problem which were not 
thought of or deemed essential to address especially 

as many wicked problems are viewed by diverse 
stakeholders who may see the problem in very different 
terms. Furthermore, it is impossible to forecast sudden 


unexpected impacts and Black Swan events. 


4. Interdisciplinary teams, composed of experts in 
diverse subjects need to work together to produce 
possible solutions. Lee Fleming demonstrated that in 
these conditions the outputs, even though not very 
relevant most of the time, have higher chances of being 
breakthroughs. For instance, missile guidance systems 
used by the United States Army in the Sixties were 
invented during the Second World War by a team 
composed of Hedy Lamarr (an actor) and George 
Antheil (an avant-garde composer). Their technology 
became the precursor to modern radio standards, such 
as Wi-Fi and Bluetooth. 


5. Providing forecasts to policy-makers, innovation 
units and public and private institutions will help them 
to formulate new social, economic and management 
policies. These new policies, in turn, will change the 


future, wearing away the forecast's accuracy. 


2. Compasses and Maps 


Generally, there are two kinds of forecasting methods: 
qualitative and quantitative [20][25]. The former is 
sometimes used if there is no data available on the 
researched topic, or if the data has little or no relevance 
to the researcher's focus. The latter is used if there is 
relevant data to assume that events from the past will 
continue to happen with little variation in the future. 
When using future forecasting techniques designers 
overwhelmingly base their methods on qualitative 
approaches and when data is used it is generally adopted 
to provide starting points and provide a semblance 

of rigor from which to base forward looking creative 
processes. It is much less evident that designers employ 
data and quantitative methods in partnership with 
qualitative throughout the wicked problem process in a 


way that each provides a feedback loop for the other. 


1 Term indicating situations and events which appear in a random 
and unexpected fashion, such as the 2008's financial meltdown 


and the dot-com bubble in 2001. 
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2.1 Qualitative Future Forecasting 


Qualitative future forecasting is based on individual 

or group creative methods focused on the problems 
addressed. Due to the high level of experiential 
individual and group based focus this method is often 
referred to as ‘judgmental’ [20]. Inevitably, these 
methods are leveraged based on an 'expert' focused 
mindset. Qualitative forecasting is employed when it 

is deemed that the future will not behave in the same 
manner as the past. A good example is the Australian 
Government's decision in 2011 to enforce companies 
to use dark green on cigarette packaging, in an effort to 
increase public health by reducing tobacco consumption 
[26]. Qualitative forecasting is helpful when a large 
amount of specific and local data is scrutinized to 
discern patterns which will not be flagged using a 
quantitative approach, as they require a more nuanced 
sensibility to be recognized. For example, when a 
construction company decides which kind of housing 

it has to build in a specific neighborhood, it generally 
seeks help from an expert on the local population, to 
better understand the demographic and cater to their 
needs. Compared to quantitative forecasting, there are 


two distinct advantages in using this method: 


* It enhances the capacity to anticipate changes in 
patterns, grounded on the knowledge of experts in 
the field. 

* It gives the flexibility to use both specific, diverse and 
non-numerical sources, which possibly enhance the 
forecast's quality since statistics cannot capture all the 
localized and individual nuances which may be highly 


significant. 


Since this method is rooted in informed opinions and 
not quantifiable data, there are some caveats, vis-a-vis 
quantitative methods. A study conducted by Lawrence 
et al. [27] highlights that the information used in 
judgmental forecasts are partial and biased to a certain 
degree, since the data used as a starting point for the 
analysis is selected without specific criteria and a fixed 


probability weighting system. 


2.2 Quantitative Future Forecasting 


Quantitative forecasts are often linked to specific 
disciplines and are developed to address a defined and 


focused situation but, overall, they can be categorized as 
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Extrapolative, Explanatory or Simulations. 


Extrapolative Methods. These methods assess 

how the series of observations made until a specific 
moment in time will continue to evolve in the future. 
They strictly use information already known to the 
forecasting team, and do not help in determining what 
factors can modify or impact the observed course. 
Consequently, it is possible to extrapolate trends and 
past seasonal cycles. Usually, extrapolative methods are 
based on time-series data [28], which are used when 
trying to forecast an event which is changing dynamically 


through time [29] such as the stock value of the market. 


Explanatory Methods. These methods look 

at explanatory variables [28], such as the prices of 
specific goods, and assume that these variables have 
an explanatory relationship with other independent 
variables. This model usually includes a third variable, 
called error, which represents the randomness in the 


system [28]. 


Simulation Methods. Simulation methods are based 
on analogies: a mechanical analogy could be a crash 

test held to verify a car's behavior in an accident. A 
mathematical analogy could be an equation describing 
the behavior of a flock of birds. A metaphorical analogy 
could be the use of a neural system to describe the 
working of a computer and a game analogy is used 
when the player's interactions are symbolic for the 


interactions inside of society [30]. 


2.3 Combining Forecasts 


Combining multiple forecasts deriving from the same 
method family is a useful procedure to achieve higher 
accuracy in the overall results [31]. This process 

of combining results requires an interdisciplinary 
approach, which leverages on the porous nature of 
social networks [32]*. Ronald Burt [33] argues that 
the higher the homogeneity of thought in a group, the 


more access to different points of views and perspective 


?This figure of speech is derived from an interview with former 
CEO of the French company Rhóne-Poulenc: "le vide (literally, 

the void) has a huge function in organizations. [...] If you do not 
leave le vide, you have no unexpected things, no creation" [34] 


[35]. 


the people which are part of multiple groups have, 
empowering them to have more options to think about 
and synthesize. They can see more broadly, select data 
and insights and synthesize them effectively in their 
practice [34][35]. 


Combining Qualitative Forecasts. Thus, it is 
evident that the starting point for an effective forecast is 
the group of people which constitute the team tackling 
the issue, the stakeholders involved and their social 
capital. The complexity lies, therefore, in selecting the 
appropriate person necessary for the specific context.? 
Combining various qualitative forecasts can be achieved 
using various methods, such as the questionnaire-based 
Delphi, the more visual Futures Wheel or the better 


known and used Scenario Making.* 


Combining Quantitative Forecasts. The analysis 
has to be done in ways that ensure repeatability of 

the process, such as an arithmetic average. In fact, 

it has been demonstrated by Robert T. Clement 

[37] that the simple average can perform as well as 
more sophisticated statistical methods. The ‘magic 
number' of forecasts to combine has been empirically 
demonstrated by Makridakis and Winkler [38] when 
they calculated the error reduction of multiple forecasts 
combined: most of the error reduction was achieved 
after combining five different forecasts. A good example 
of combining forecasts is the work of Lobo and Nair 
[39]. They analyzed the quarterly earnings forecasts 

for 96 different companies between 1973 and 1983, 
employing two separate qualitative methods and two 
quantitative extrapolations to justify their projections. 
By combining the judgmental outputs, they lowered the 
Mean Absolute Percentage Error (MAPE) by a total of 
5.2% [39]. 


3. Future Forecasting for Designers 


Despite the advantages future forecasting offers in 
terms of risk mitigation, efficient planning strategies 
and preemptive decision making in the context of 
economics and business management, the concept has 
its limitations, as extensively addressed by van Vught 
[40] and Khan and Mann [41]. The pitfalls of qualitative 
methods stem from logical fallacies (that is, those which 
depend on the human factor) and are a byproduct of 


an underlying fallacy deriving from Hegel's historicist 


approach [42][43]. His arguments are grounded on 
inductive reasoning, which states that if it is possible 
to observe a certain regularity in a limited number 

of situations, it is feasible to formulate a generic 
statement which implies the future repetition of that 
particular instance [44]. This mental framework alone 
is questionable, as argued by David Hume and Bertrand 
Russell? [45]: they evidenced the complete lack of any 
logical argument on which it is possible to base the 
assurance that future experiences will resemble in 

any way an already experienced phenomenon. On the 
other side, an approach to future forecasting which 
relies solely on computational and quantitative methods 
can impoverish the design activity by reducing the 
ability of abductive reasoning to identify alternative 
futures, in effect it can radically reduce divergent 
thinking [46]. An improved mental framework for 

the designer can be suggested by the mixed use of 
quantitative and qualitative approaches. If quantitative 
and qualitative research is seen as a continuum, where 
qualitative-driven research is mainly "constructivist- 
poststructuralist-critical" and quantitative-driven 
research is post-positivist [47], the mediation of the two 
is given by pragmaticist and pluralist approach to the 
future [48]. 

To summarize we have argued that traditional design 
approaches to future forecasting are under-powered 
in not fully leveraging the capabilities of quantitative 
methods integrated throughout the creative process 
of addressing wicked problems. Existing methods 
from speculative critical design are powerful ways of 
visualizing alternative futures but have weak feedback 


loops and were into originally intended to tackle wicked 


?|t is important to stress that social capital is not a prerogative 
of a selected group of people, such as lawyers, scientists or 
politicians: it lies in those structural holes in the fabric of society, 
as Burt describes them [38][39]. 

^ For a thorough review of qualitative forecasts and how they 
can be combined, see Glenn & Gordon's Future Research 
Methodology [36].° Russell made evident the fallacy of inductivist 
reasoning with the famous chicken example: "We know that 

all these rather crude expectations of uniformity are liable to 

be misleading. The man who has fed the chicken every day 
throughout its life at last wrings its neck instead, showing that 


more refined views as to the uniformity of nature would have 


been useful to the chicken". 
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Fig. 3. Diagram illustrating the main interacting elements of the proposed new 


design-led future forecast framework for tackling wicked problems. 


problem directly. 


3.1 A New Framework 


We propose a new framework for addressing wicked 
problems using mixed quantitative and qualitative 
methods leveraging creative design methods. The 
framework has a number of interacting elements 
initiated via first identifying a gap in capability for 
tackling a wicked problem. Gap analysis is a term 
borrowed from William Burley's research in biological 
diversity [49]. He defines a gap analysis as a simple 
concept, which can be described as a continuous loop 


between: 


1. Defining and classifying various elements of 
biodiversity in a specific area; 

2. Examining the preservation systems already present 
in the territory; 

3. Determining which elements are either 
underrepresented or absent from the initial assessment; 
4. Setting guidelines for the next conservational effort 
[49]. 


This could be transposed in the design practice by: 


1. Defining any number of qualitative future trajectories; 
2. Quantitative analysis of datasets and determining 
trends stemming from it; 

3. Determining the discontinuity elements between the 
two; 

4. Setting design principles and constraints for the 


project. 
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The diagram in Fig. 3 structures the main elements 
of the proposed new framework which functions as 


follows: 


The framework proposes an approach to wicked 
problems from a mixed methods perspective via 
employing quantitative, qualitative and combinatorial 
methods to future forecast thereby increasing the 
forecast's accuracy and its communicability to a 
variety of involved stakeholders. The synthetic 
process proposed consists of a gap analysis, where 
different future trajectories deriving from the different 
methods employed are mediated by individuating any 
discrepancies resulting from the different processes. 
The envisioned design solution or intervention is then 
steered continuously by the feedback loop given by its 
own impact on the problem in question via qualitative 
and quantitative data and insights .This framework is 
proposed explicitly for designing and wicked problems; 
therefore, it is intentionally not structured as a series 
of steps to follow and implement: depending on the 
problem at hand, the stakeholders included, the 
available datasets and the level of access to experts in 
the field, different processes and outcomes are to be 


expected. 


Further research which grounds the framework in a 
specific wicked problem is needed, so that it can be 


adapted and tested in a specific instance. 


4. Conclusion and Future Steps 


Introducing the proposed qualitative-quantitative 


framework for future forecasting aimed at tackling 


current and emerging complex challenges seems to be a 


promising step towards the development of a systematic 


method for resolving wicked problems. Reflecting 
back on the definition provided by Rittel and Webber, 
it becomes clear that the use of mixed techniques 

of future forecasting could be a desirable method to 


scan the boundaries of a given problem initiated via a 


combinatorial strategic design approach. However, since 


future forecasting is often considered to be an aleatory 


art without the ability to define the future, its task 

is not to dictate specific lines of action, but to chart 
possible outcomes. Therefore, future forecasting as 
presented in this paper is another method available to 


the designer to systematically create routes to action. 


One of the challenges for our proposed approach is 
the dominance of qualitative approaches in design and 
the mistrust, misunderstanding and lack of exposure 
to the value of quantitative methods. Moreover, many 
designers consider data driven conclusions that hold 
central truths to be illusory. While we believe there 
is significant value to considering this approach, the 
disciplinary methods preference in design is also 

a serious consideration. We have also highlighted 
the inclusion of some cybernetic concepts including 
feedback loops and there is also potential in further 
exploration of the relationship between wicked 
problems and black box technologies and how these 
problems could be addressed by a new category of 
mixed methods for future forecasting incorporating 


qualitative and quantitative feedbacks. 
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Eventual Design for an Emergent World 


Abstract 


Are we at a punctuation point in the evolution of design 
practices? The ascendance of modern design practices, 
over the last 100 years, to popular and pervasive 
application, is fraught with questions about benefits 
and limits. In light of persistent social injustice, political 
strife and looming ecological danger, a more extensive 
accounting of the work of designers is needed, if only to 
craft a proper manifesto for proceeding with our work. 
By extrapolating recent trends in the evolution of 
design practices, a meta-practice of “eventual” design 
is envisioned, to call attention - beyond form of artifacts, 
process of service, and quality of experience - to the 
entailments, actions and etiquette of events, and 

their eventual consequences. The “event” is identified 
as a fruitful entry point for forensic reconnaissance 
into causes, effects and consequences in this terra 
incognita. An event lab and instrument are sketched, 
with intentions to 1. identify aesthetic causes, social 
effects and systemic consequences of design; 2. 
formulate a common etiquette between and amongst 
people and with technology; and 3. model and monitor 
interdependence and sensitivity within the logistics of 
our human project and ecology of living systems to 


avoid detrimental or disastrous consequences. 


Keywords 


Design, Consequence, Events, Aesthetics, Etiquette 


1. Introduction 


Design, in essence, is a kind of wishful thinking 
combined with audacity and ingenuity. As designers, 
given any evidence of the worlds we inhabit, we can, 
with classic “blues” mentality, recite a list of ills, flaws 
and shortcomings that dismay. So, when we contemplate 
a new approach to design, consistent with our times, it 
is instinctual to not look away from past disasters and 
present dangers of human making and doing. But it is 


axiomatic that we imagine optimistic goals. 


Design is also a kind of guessing game in which the 
guesses include ideas about the remote existence of 
the “other”; ideas that are presumptuous and biased 
by personal experience; ideas that use anecdotes and 
asymptotes to conjure archetypes and prototypes with 
impunity, and are realized with a peculiar ingenuity and 


determinism. 


Design is also often conflated with deciding. Origami 
provides a clarifying metaphor: While folding moves 
determinedly toward an explicit exclusive formal 
expression by deciding, designing opens new possibilities 
of form, by cutting. Design prototypes first subvert and 
violate the norm, before assimilation of the preferred is 


replicated. 


While design is often cited as a means to goodness, 


beneficial change, and betterment of life, there is a 
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dark side to this noble tradition; this speculative studio 
practice of intentionally making mistakes and fixing 
some of them; this mutative operation with putative 
benefits. It seems imperative, as we proceed, to not 
ignore the penalty of misunderstanding each "other" 
and conflicts inherent in the rise of a single species to 


dominate nature with "human" nature. 


Modern design principles have had tremendous 
influence over the last one hundred years on most 
design practices and arguably constitute a major cultural 
force shaping infrastructure and implements of the 
modern world. But as demands and desires of one 
species threaten the existence, natural resources and 
habitable environment of other species, we risk epic 
eventual calamity. Philosopher Stanley Cavell said it well 
as "our capacity to do harm has exceeded our capacity 
to do good.” [1] 


Designers and their clients, largely absolved of liability, 
are nonetheless instrumental in delivering this moment. 
Even conscientious designers invoking methods of 
social and environmental science in their research 

lack sufficient means to understand or account for 
derivative, long-term, far-reaching or unintended 
consequences. The question goes begging: How should 
we account for the shortcomings, limitations and 


constraints of design practices? 


Designers have been, and remain, active and integral to 
capitalism, industrialization and globalization. Flaws in 
capitalism, however, exacerbate disparity and rupture 
the fabric of everyday life. Excesses of human habit and 
density of human habitat proceed toward depletion and 
saturation of our ecology. Corruption and regulation 
distort and circumvent intended results. Design 


happens. 


Sublimated violence inhabits our infrastructure. Errors 
that can rapidly propagate unknown risks, persistent 
threats and real danger to living systems have a 

new velocity: the speed of light. Digital connectivity 
accelerates toward ubiquity throughout the logistics of 
our human project. Interdependence amongst humans, 
and with and within technology, grows, both tantalizing 


and ominous. 


In the late 1980s, the entire national 
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telecommunications infrastructure in the United States 
faced total collapse from a cascade of errors spawned 
from a single typo in one million lines of code. How 
will we avoid catastrophes of such proportions as 

the cybernetic future expands like the universe and 


evidence of its constitution fades? 


Every single material object belies the violence of its 
origins, its production and eventual dangers of its 

use, abuse, distribution, abandonment and ultimate 
interment. Manifest, latent, buffered or muted, violence 
constitutes a real terror of our existence and needs a 
remedy. Is there a link between error and terror we do 


not see? 


Questions face designers (and their clients) as to 
whether existing design processes can be applied to this 
emergent world or even address failures from the past. 
Can we better understand the social and ecological 
consequences of "designing" the world? Can we resolve 
this emergent crisis with social justice or ecological 
reciprocity? Can we even eliminate or mitigate violence 
with acts of human kindness? More crucially, can 
designers address how people treat each other and 

our living ecology or will we continue to primarily 
serve a mass marketplace of individual wishes? If we 
are at a punctuation point in evolving design practices, 
can designers adopt new goals that subvert their own 


process, progress or success? 


We are Much Obliged. Is it too late to accept the 
greater role of citizen designer exemplified by the 
utilitarian legislator Benjamin Franklin, and too soon 

to be the universal citizen Buckminster "Start with the 
universe" and “If the results are not beautiful, | know | 
am not done" [2] Fuller? Now is always the opportune 
moment to crack the code of design, reveal its potential 
and reform our praxis. As anyone who has planted a 
tree knows, the best time to plant a tree is twenty years 


ago. 
The second-best time is now. 


2. Manifesto Destiny? 


The recent surge in popularity of design has prompted 
multiple declarations of the power of design to produce 


beneficial change and one noble attempt to write a 


manifesto to enact the idea that "All people deserve 
to live in a well-designed world." [3] Design principles 
and practices are being celebrated with an invitation to 
"prototype the future" [4] and dozens of exhortations 
to innovate, disrupt and change the world. "Utopia 
Toolbox" [5] catalogs an illuminating collection of such 
ambitious initiatives. Even the United States Army has 
incorporated a doctrine of design into its arsenal of 
weapons to wage persistent war. "In a dynamic and 
multidimensional operational environment, design 
offers tools vital to solving the complex, ill-structured 


problems presented by persistent conflict." [6] 


A Bit of Personal History. Fifty years ago, as 

an impressionable young design student, | spent an 
inspiring day with R. Buckminster Fuller who told us the 
goal of design was to “make the world work for 100% 
of humanity, in the shortest possible time, through 
spontaneous cooperation without ecological offense or 


disadvantage of anyone." [7] 


We clearly have yet to accomplish that goal. 


Within short months of my infection with Fuller's 
enthusiasm, | was drafted into the army, ostensibly for 
the vile war in Vietnam. | rapidly learned some hard 
lessons about freewill, individual agency, large-scale 
forces in lethal conflict, destructive power of well- 
designed weapons and above all, a spectrum of human 
behavior well beyond my childhood experiences. 

Even an aesthetic difference, between the hard kick 
from the recoil of an M-14 rifle and the easy burping 
nudges from an automatic M-16 rifle, signaled a 
profound change in my possible relationships to 
other humans; that signature of a new tragic sense of 
"self," a “medium,” and a realization of the recondite 


remoteness of the "other." 


Something happened. Something broke within me. | 


mutated. 


As a consequence, | became a cynical pragmatist, 
maybe even a realist. My activism tempered, my idealist 
fantasies buried, (but latent in every subsequent design 
consideration) | became host to perhaps an ideal 
mental brew for pursuing the intelligent response to 
uncertainty and opportunistic risk management that 


design, of necessity, becomes. 


Design Practices Evolve. What was, in the 19th 
century, a choice between utility or decoration 
became, in the 20th century, a focus on synthesis of 
form and function. By the 21st century, digital tools for 
design were amplified by revolutions in computing and 
communication capacity. A fascination with interaction 
and distribution erupted, permitting designers to 
develop integrated systems of products and 

services for "personal" experiences through media 


design practices. 


Since, as Eva Horn has written, "Media are not only 
the conditions of possibility for events—be they 

the transfer of a message, the emergence of a visual 
object, or the re-presentation of things past—but are 
in themselves events: assemblages or constellations of 
certain technologies, fields of knowledge, and social 


institutions." [8] 


Old, New (Now), and Eventual. Two lists (see 
below) of trends in design practices, shared with 

me by Hugh Dubberly, were the impetus for my 
speculations on design practice over a decade ago. 
Extrapolating emergent aspects of design from those 
two lists spawned the notion of the "eventual". The 
question of what happens as a consequence of what a 
thing is and what it does, provoked the idea of turning 
the attention of designers from objects to events to 
better understand cause and effect, especially in social 


behavior. 


In the old dominant design paradigm, heroic solo 
product designers worked on the composition of 
editions of static objects in a fixed locus (node) using 
information drawn from computers. They sought 
simplicity, if not for the sake of the relationship of form 
and function, at least to make manufacture for massive 
consumption efficient. Decisions and control followed a 


top-down hierarchy. 


In the new (now), then dominant, design paradigm (in 
the golden age of the Internet and web) teams of service 
designers used improvisation and continuous updating 
of dynamic links to invent experiences using persuasive 
communication. They embraced and exploited 
complexity, if not for the sake of tighter relationships 

of form to belief, at least to enrich interaction and 


fascination with screens. Decision-making and control 
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Old New (Now) Eventual 
Product as object experience behavior 
static dynamic liberal (as in liberating) 
node link mesh 
Focus on information persuasion transformation 
computing communicating conversation 
with through within . . 
Key skills manufacturing interaction design etiquette design Table 1: Terms thefirst threecolunns 
product design service design social networks (below) are from a memo about design my 
Process seek simplicity embrace complexity achieve sophistication colleague Hugh Dubberly wrote. They compare 
solo or hero team community . . 
top-down evolutionary metabolic Old and New aspects of design. Extrapolating 
editions continuous updating anticipatory from those two lists and integrating the results 
yielded my initial notions of eventual design and 
Technique composition improvisation performance 


were an evolutionary collective process. 


My wishful emergent paradigm proposes, that 
communities of etiquette designers would use 
cybernetic conversation to instigate liberal (liberating) 
behaviors within a mesh of social networks. They would 
embed sophistication in their design of etiquette, if not 
for the eventual goal of transforming living systems, at 
least to improve relationships between semiotic form 
and social performance. Anticipation would supersede 
decision-making and control would yield to contextual 


eventualities and metabolic entailments of events. 


Events, Appearances, Performances. Slavoj Zizek 
postulates an event as when “effects appear greater 
than apparent causes.” [9] An eventual design practice 
would juxtapose events to consider long and short-term 
causes, effects and possible consequences, not simply 


what is or can be. 


Events disclose how people choose to invest their 
attention and how that attention turns into obligations, 
agreements, appointments, and thereby into actions 

of life. The salient design question is, “...and then 

what happens?” Events also significantly occur at and 
comprise the intersection of information design and 


experience design practices. 


Turning Attention. Can designers invent and deploy 
a "macroscope" [10] to reveal and mitigate inherent 
conflicts between nature and human nature? Can the 
cultural currency of attention find parity or equity 
with the social currency of trust? Can we conceive 


and design a common public etiquette for civil society 


38 Design and Semantics of Form and Movement 


etiquette. 


to comprehensively emerge? Can social or ecological 


justice even be design goals? 


It is now possible to trace the events of media from 
sources to uses and track media incidence to individual 
experience. Behavioral insights from data-mining permit 
design of systems to surveil, anticipate, inform and 
influence individual experience. It may now be possible 
to assess the impact media have on relations between 


people. 


It should be possible for designers to deploy diagnostic 
and forensic design research as a reconnaissance of 
new cultural territory to learn how the spectrum of 
etiquette between and amongst people can become an 
index to social justice or design errors. Perhaps this can 
extend to measuring our collective treatment of our 


planet's ecology? 


It is just barely imaginable that such sophisticated 

design of media could ultimately engender sustainable 
beneficial relationships between and amongst humans. 
However, it does seem feasible to move the attention of 
designers from the millisecond events between a person 
and their instant messages, games and signals to other 
events occurring between and amongst people and to 
potentially map the consequences of human choices and 
actions to living systems and nature over deep time. 
What other success could be more meaningful to civil 


society? 


3. Etiquette 


How we behave toward others is a spectrum that 


ranges across confluence, collaboration, cooperation, 
coordination, competition, conspiracy, conflict and 
coercion. Perhaps a spectrum from fear to love can 
serve as an index to human treatment of our ecology of 


living systems; our global common ground? 


The writer Guy Davenport recalled Jane Kramer saying 
that the French "preferred a common etiquette to a 
common ground." [11] This comment crystallized the 
idea that social justice in denser diverse populations 
will require a more sophisticated understanding of the 
"other" and how to integrate valuable differences in 
each "other" into society. Adjusting our design scope 
to examine events opens a portal into how form 

and significance pass across social synapses and have 


subsequent or derivative consequences. 


Civil society requires, or rather, obliges each individual 
to acknowledge the value and protect the freedom of 
the "other." An understanding of "kind" and a common 
etiquette of kindness is required for negotiation and 
integration of valuable differences to replace strife and 
conflict. The real mandate for change of design practice 
comes from accepting this moral accountability for 
consequential events of design decisions. Can designers 
seek such a noble goal when elements of living systems 


are so complex? 


4. Framing, Forming and Informing a New 
Design Practice 


Conceptually framing the study of events and the 
eventual can begin with the "welkin" - the sky we share. 
Under that sky, the watershed of earth symbiotically 
grows fuel for life. The equation of life we work within 
is reciprocity or kinship between logistics of the human 
project and ecology of living systems. At heart, our 
concern is action and reaction, between people, and 


between human nature and nature. 


Do we control that symbiosis or will technology beyond 


our intelligence intervene and mediate living systems? 


Logistics. All of human history can be described asa 
logistical project to support the growth of the human 
population. An archaic and useful definition of logistics 


as "what must be known for the conduct of war" [12] 


helps us see that human project as intrinsically lethal 
and violent. How we prosecute the human project could 
be described as a kind of war against nature and struggle 
against natural forces. But it is actually within us, within 
the context of evolution. Will we govern our logistics by 
design, by sentient technology or have we lost even the 
capacity to govern technology by logic, or ourselves by 


law or common sense? 


Ecology. Our common ground is finite. Our capacity 
for meanness and kindness seems unlimited. Witness 
the plight of indigenous hunter-gatherers whose lands 
are confiscated with the rise of the nation-state. They 
occupy boxes of concrete that are hot in the summer 
and cold in the winter. With the arrival of subsidized 
and mandated provision of electricity, they obtain 
refrigerators. With refrigerators, they obtain soft 
drinks. With a diet of soft drinks, they obtain diabetes, 
further decimating their population. This in all of ten 
years out of thousands living sustainably. The spectrum 


of etiquette includes ecologic cruelty. 


Kinship. When James Grier Miller, instrumental in 
formulating behavioral science, wrote "General Living 
Systems Theory," he was inviting the social sciences 

to find a way to join with the natural sciences to 

better "understand the facts of life." [13] He invoked 
cybernetics as a tool for accomplishing that and, more 
importantly, identified and defined a systems approach 
to the natural ecology that permitted examination of 
how society might achieve civilization in concert or 
kinship with nature; a kinship monitored within events. 
Kinship and Kindness. Kindness is a social solvent. 
Kindness also requires a study of difference. To 
integrate valuable differences in situations or events 
one must understand kind. Kindness may turn out to be 
crucial to the survival of the fittest when we consider 
our social interdependence. Behavioral economics might 


explain our neglect. 


Aesthetics. Beyond physical reflexes, humans have 
evolved and embody a system of signs, the semiotics 

of everyday life, as a counterpoint to the physics 

of everyday life. These signs mediate between the 
metabolic and cybernetic systems which govern 
everyday life. The aesthetic goal for individual and social 


experience would be for events to dissolve into timeless 
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moments. For this, we need to better understand the 


link, the kinship, between aesthetics and etiquette. 


Timebase over Database. As a time-based meta- 
practice, eventual design would work by coordination, 
configuration and sequencing of events; with 
temporalities, appointments, assignments, and notions 
of subjective and elastic time. All of the qualities and 
paradigms of time delineated by Olga Ast in "Fleeing 
from Absence" [14] enter into consideration. Events can 
act as armatures for prototyping ideal living systems, but 


how can we account for all the entailments of events? 


Technological Feasibility. Bruno LaTour said it: 
"New innovation will be absolutely necessary if we 

are to adequately represent the conflicting natures 

of all the things that are to be designed...visualization 
tools that allow the contradictory and controversial 
nature of matters of concern to be represented." 

[15] At a minimum we might reform architectures of 
computation and communication to employ parallelism, 
connectome mapping, record/playback, fuzzy and 
three-state logic, data valencing, metabolic processing, 
photonic switching, fractal zooming, depiction 
morphing, consideration spacing and pacing, sociocratic 
programming, blockchain accounting, link typing, 
quantum computing and sensitivity modelling in an event 


laboratory larger than the experiment. 


Temporal Nominal Annotation. The construct of 
the tenome | imagine could begin to monitor a temporal 


membrane across which stories are told, lives are lived, 


Event Markers and Maps 
Second 


and histories are narrated. While the genes and memes 
circulate in matter and memory, their intersection in 

time as social interactions might be marked in a tenome. 
A carefully annotated tenome could locate and score the 
interstitial event, where media disappear, and exchange 


of cultural and social values takes place. 


The site of exchange could even permit value to be 
extracted by a mediator, the way a bitcoin miner is 
compensated. Knowing what to do next; the etiquette, 
[16] or reading choices offered by the "ticket" could 


emerge from data within the tenome. 


Templates of tenomes (see below) would be a possible 
way of marking time, annotating situations and mapping 
scenarios for deliberation, analysis and sharing insights. 
(Binary systems, which now tend to "eat their young," 
should evolve such consideration space/time to more 


naturally integrate with lives, as lived, of human beings.) 


For Example. When the replacement of the decayed 
planks of a park bench is connected to a decision to 
plant a tree decades in advance, and the construction 
of the bench permits this repair to be a social event; 

a sense of continuity and community ensues; a natural 


order is preserved and enjoyed. 


When that kind of knowhow and understanding 

is conveyed over generations, a sophistication 

is embedded in the designing and deciding that 
subdues violence. A pragmatic, yet noble transaction 


occurs when preparedness has replaced planning 


Cycle(s) Per 


EE TL LLL ALLL ALL, 


Minute 
Hour 
Day 
Week 
Month 
Year 


Decade 


Individual A mee 
Individual B mee 
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Life 


Tenomes 

are frames twisted 
to reveal time 
cycles. 


Timeframes 
are lists to make 
entailments of 
events legible. 
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Events as Potential Moments 
Marking direct and mediated interactions between people, media and machines. 
Locating consequences, origins, causes and effects. 


Figure 1. Schematic of 
a method for marking 

a time-base as events 
between sender and 
receiver. Entailments of 
the event listed, and time 
cycles twisted into time 
frames assist in seeing 
chronology, sequence 
or series. Parallelism 
would reveal potential 


correlations. 


and the eventualities of nature are respected and 
accommodated. The persistent practical question is 


simply, "...then what happens?" or “Great, now what?" 


5. The Goals of Eventual Design 


Eventual design is proposed as a meta-practice to 
address situations of human experience between and 
amongst people within the logistics and ecology of an 
emergent world. Its praxis is a holistic investigative 
approach to design of mediating form, that begins within 
an imaginary, but necessary, School of Behavioral Arts, 
where all disciplines or practices exploring, employing 
and deploying the human body convene. There, a study 
of form and function, form and belief and form and 
behavior, seeks to understand life as a performance and 
discover essential, optimal and eventual relationships 


between form and fruition. 


Identifying. Events link cause and effect. Events hold 
clues to quantum entanglements. Events signify and 
illuminate life. Events, from geologic to nanosecond, 
mark time. Events convey cultural signals. Chains of 


events link deep history, inheritances and influences. 


Mediating. Form affects behavior. Designing form 
that mediates between people can establish and enhance 
etiquette. The test? If the interface, artifact, experience, 


behavior feels rude, it is rude. 


Monitoring. Relationships between the logistics of the 
human project and the ecology of living systems over 
deep time can be registered as interdependent and 


assessed for consequence, and eventualities located. 


When in the Course of Human Events. Fuller 
likened design to electricity, writing “We cannot define 
it, but we can measure its effects." [17] Yet despite 
recent advancement of design research, we seldom 
measure effects or aftereffects of design, nor do we 
have useful metrics for effects we observe. Instead, we 
rely on market forces to reward and punish the work of 


designers. 


The emergent situation of the present world poses 
challenges neatly articulated by a truly global explicitly 
and implicitly violent force, the U.S. military as: 


"Globalization, technological diffusion, demographic 


shifts, resource scarcity, climate changes and natural 
disasters, proliferation of weapons of mass destruction, 
and failed or failing states." [18] Add density of human 
habitat, and ubiquitous digital connectivity to compress 
reaction times and accelerate recoil. Any design 
renaissance that alleviates yet untreated problems of 
the planet will require a sophisticated reconnaissance 
of actual and potential consequences. A quantum shift 
of attention from object to event, from information 
database to human time-base, offers a way to seek 

and evaluate such correlations, causes, effects and 


consequences. 


You are Invited to an Otherwise Future. Ideally, 
designers would develop formal systems that advocate, 
enable, distribute and sustain social justice; where 
inherent conflicts of human nature within our common 
ground are resolved; where the form and content 

of infrastructure, artifacts, and appliances establish, 
inform and evolve a common universal etiquette that 
reconciles human nature with other wisdom. The 
daunting complexity of living systems, our growing 
interdependence as humans, and with technologies 
that animate and emulate life, constitute our event 


laboratory. 


| cordially invite you to enter. 
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The Decentralization Turns in Design: 
An Exploration Through the Maker 


Movement 


Abstract 


This article recollects paradigms and turns in the Design 
research and practice integrating several contributions 
and elaborates a new proposal arguing that the current 
changes are not about a single shift from one center to 
another, but rather the decentralization and distribution 
to several new centers becoming thus a system of 
Decentralization turns in Design. This system is 
structured with eight turns: Do It Yourself approaches 
to Design, Open Design, Network Science approach to 
Design, Distributed Manufacturing, Design and Locality, 
Design and Decolonization, Posthuman Design, Design, 
Data, Software and Artificial Intelligence. These eight 
turns are transforming design practice and research 
along eight dimensions (meta, who, what, where, 

how, tools, process, scale), here explored through the 


context of the Maker Movement. 


Keywords 


Decentralization, Design Turns, Maker Movement, 


Posthuman 


1. Introduction 


The understanding of the evolution, revolutions, 
general dynamics and changes in the history of science 
is a crucially strategic endeavor. In this sense, one of 
the most influential contribution was elaborated by 
Kuhn [1], who theorized that science evolves through 
revolutions based on paradigm shifts that move the 
focus from one center to a new one, transforming how 
science is done and understood and generating new eras 
in science. Besides epoch-making paradigms science 
presents also turns, which are smaller-scale changes 
that take place more as parallel and overlapping [2, 3] 
and can be considered a product of their own time. At 
different levels, decentralization is a crucial element of 
the history of science. This paper aims to contribute 
to this discussion regarding Design, by tracing a 

first exploratory timeline of paradigms and turns in 

the design research and practice, and by identifying 

a further turn that takes place now. Therefore, it 
proposes the extension of these turns not by showing 
the displacement from one center to a new one, but by 
distributing the center to several new centers in what is 
can be then described as a set of Decentralization turns 
in Design. Moreover, the identification of such new 
turns is based on eight turns bringing decentralization 
to design along eight dimensions. Although these 
dimensions have several different roots and also 


applications, this paper describes them in the specific 
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context of the Maker Movement [4, 5]; as one of the 
more recent evolution of the Design practice and with 
extended implications, it offers examples for all the 
identified turns. Based on this context, this paper aims 
to answer the following research question: What are 
the decentralizations that have been emerging in the 
design research and practice, and how they can possibly 
characterize new turns in Design? In order to answer 
that, this research explores with a literature review 
the previous turns in design, and elaborates a coherent 


system of turns in Design and in the Maker Movement. 


2. A Review of Paradigms and Turns in 
Design 


The identification of paradigms and turns in Design 
might have less tradition than in other disciplines. 
While some authors provide an overview of timelines 
of different paradigms and turns, others develop single 
turns that show their underlying research and practice, 
exploring the role of designers and users, and also 

of the methods and theories applied. Such paradigms 
and turns present changes in design practice, roles, 
processes and concepts, and we focused specifically 
on the proposals that focus on agency by shifting who 


design and how. 


One of the main contributions in the identification of 
paradigms in design can be found in the four economic 
paradigms identified and described by Gardien, 
Djajadiningrat, Hummels, and Brombacher: the 
Industrial, Experience, Knowledge and Transformation 
economies [6]. This framework is mainly focused on 
companies and how they can innovate in order to keep 
up with paradigm changes. In the Industrial Economy, 
mass production and a modernist machine aesthetic are 
based on efficient design activities that are objectively 
measurable and follow a linear rational problem-solving 
process. In the Experience Economy, differentiation is 
created through branding as a way to enable users 

to belong to subcultures with a plurality of diverse 
styles. The work of designers here extends beyond 
object-centered thinking to the total user experience 
and follows two approaches to processes: reflective 
practice and user-centered design. In the Knowledge 
Economy, an overflow of brands and the emergence of 
the remix culture of online communities enable users 


to reach self-actualization through a DIY remix of 
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subcultures into individual custom subculture: belonging 
happens through a personal remix of one own's identity. 
Companies create value through an open innovation 
process based on communities of users with an open 
design, co-created and participatory approach. Now the 
works of designers is in orchestrating open innovation 
through algorithmic thinking, data visualization and 
creative coding. The Transformation Economy is the latest 
and still emerging paradigm, where concerns about 
pollution, global warming and wealth disparity push the 
work of designers towards understanding societal values 
and foster reciprocal value creation with a systemic 
approach to local solutions through platforms that 


enable plural and radical approaches. 


In terms of turns, although design researchers have 
already identified turns in design in the past [3, 7], there 
is not a consensus in their definition, identification, 

and meaning of the word "turn" in the design field. 

We consider a starting point for the study of turns 

in design the paper of Marttila and Botero [3], who 
identified four turns related to "co" in co-design: 
Usability, Sociability, Designability, and Openness. 

For them turns are more parallel, overlapping, mixed 
and connected than paradigm shifts. In other words, 

a turn is a device for communicating the identified 
changes. The Usability turn can be considered the first 
turn in Co-design, and it has a clear emphasis on the 
user and the use situation. It provides an impulse to 
the development of the concept of Human Centered 
Design (HCD) and constitutes the basis of research and 
literature in Human-Computer Interaction (HCI) [8]. 
Here, the user can be described as a consumer, and the 
primary purpose of design is to make them consume 
the final result of the design process. The second turn 
is described as the Sociability turn and it is characterized 
by the attention to the relationship between peoples' 
practices and to facilitating stakeholders' contributions 
[3, 9, 10]. Moreover, the Sociability turn defines design 
collaboration as enacted through organized events (e.g. 
workshops) initiated by experts, in which users are 
considered stakeholders who are not merely passive 
consumers, instead they become more active and 

form partnerships [11], supporting the development of 


practices like co-design and participatory design. 


Therefore, Design acquires a broader notion, involving 


activities that support people's wishes to become 


active participants and contributors in personally 
meaningful activities [12], which characterize and 
develop the Designability turn. Moreover, users need 

to be empowered "to act as designers" [13], which is 
related to the concept of Meta-Design. In the Meta- 
Design approach, users are seen as potential designers 
by extending, improving and appropriating design, 

while designers are the ones in charge to create social 
and technical infrastructures to enable new forms of 
collaboration. This differs from the Openness turn, that 
is solely based on the adoption of digital technologies 
and the rise of the Open Source Movement, enabling 
new forms of organization and distribution of resources 
through sharing and collaboration [14]. New spaces 

and conditions for practice can be developed regarding 
the openness of projects, the rise of new practices and 
the sharing of knowledge and information, enabling 

new forms of collaborative interactions. Moreover, it 
allows end users to share projects and access to digital 
fabrication technologies to prototype and manufacture 
products locally [15-17]. The openness of design 
changes and challenges some of the current paradigms 
of design practice, such as: authorship, and the idea that 
every design project should end with a commodified and 
final outcome. Two main strands can be identified in the 
practice and literature on Open Design: a predominant 
one focusing on design artifacts in which the emphasis 
is on the openness of publicly available designs (e.g., 
blueprints as documents), and a broader approach to 


open-ended design activities and practices. 


Other authors propose single turns instead of a timeline 
of turns, such as the Semantic, Openness and Peer- 
to-Peer, Systemic and Posthuman turns. Krippendorf 
[7] describes the Semantic turn in design as a shift in 
designing artifacts that moves the center from how 
artifacts ought to function to what they mean to those 
affected by them: the design of an artifact entail more 
than just its form and aesthetic, it is also connected 

to users' beliefs, values, needs, and emotions. The 
Systemic turn is distinguished from others in terms of 
scale, social complexity, and integration. By integrating 
systems thinking and its methods, this turn extends 
Human-Centered Design towards complex and multi- 
stakeholder service systems [18], an aspect that is also 
related to the idea of layers and interdependence of 
different turns. In the Openness and Peer-to-Peer turn 
Menichinelli adds the Peer-to-Peer (P2P) and Meta- 


Design approaches to the Openness turn, considering 
how design can adopt open source and P2P practices 
and how it could use its tools, practice and resources 
for implementing open source and P2P practices [19]. 
Within the context of this research, this turn becomes 
relevant since it openly proposes decentralization as 
both an explicit core value and objective. The meta- 
design approaches detailed in this turn explicitly use 
design tools and approaches not just for operating 
within decentralized systems, but also for creating 
them. The importance of decentralization is here 

an inheritance of the origin of these phenomena 

from Internet-based organizations and its historical 
decentralized architecture elaborated by Baran [20]. 
In the Posthuman turn described by Forlano, design 

is a way to resist to binary categories and integrate 
humans and the non-humans as social constructions, 
which is connected to the development and use of 
new technologies and a new understanding of agency 
[21]. Posthumanism criticizes the anthropocentrism, 
extending the limits of the human also to the ethical, 
social, and political sphere in which humans operates. 
The agency of artifacts distances the centrality of 
humans in design research and practice, moving towards 
a more complex system of interactions between human 
and non-human actors. Moreover, this comprises how 
socio-technical systems can be socially constructed, 
but also how they can shape society. Therefore, not 
only it discusses the interactions between human and 
non-humans, it is also a way to break the notion of 
subjectivity (subjectivities and identities constructed 
through race, class, gender, sexuality, and ability) in 


Design. 


3. The Decentralization Turns in Design 
and the Maker Movement 


Several turns in Design have emerged in the last 
decades, sometimes overlapping and sometimes 
disconnected among them. Some of these turns are 
now increasingly integrated because of the recent 
technological, social and economic changes, bringing 
an additional displacement to the ones documented 

in the previous section. To better understand the 
possible changes and opportunities that can be related 
to decentralization, as an integrated and transitory set 
of turns, we analyze how this unfolds in the relation 


between the Maker Movement and the Design practice 


Design and Semantics of Form and Movement 45 


and research. In this context, the Maker Movement is 
not seen as the only cause of this new turn, but more as 
one of the causes and one of the effects of such turn at 
the same time, both an outcome and a catalyst of these 
changes, and the context that inspired this research and 


the reflections. 


Through this analysis, we aim to identify what are the 
clear forces towards the decentralization in design, 
divided by eight turns and eight dimensions in which the 
effects decentralization takes place. We propose to call 
it as Decentralization turns, as it represents a clear force 
towards decentralization, multiplied by the integration 
of several turns and the extension of them. The deep 
changes behind this turn are aligned with the evolution 
of the design research and practice, which has recently 
moved its scope from single users to local and online 
communities, from isolated projects to complex system 
of solutions, and from harmful and misunderstood 
impact on the environment to a long-term, inseparable, 


aware and unavoidable blending with it. 


We argue then that this is not just a Maker Movement 
turn but a thorough, complex and articulated 
phenomenon. Therefore, the Maker Movement and 

its integration with Design research and practice can 
be where all the recent decentralization processes 

are meeting, showing that there are wider and more 
profound implications than just the use of 3D printing 
technologies and informal education and design. Within 
this context, the system of Decentralization turns in 
Design can be identified in eight distinct categories: 1) 
Do It Yourself (DIY) approaches to Design, 2) Open Design, 
3) Network Science approach to Design, 4) Distributed 
Manufacturing, 5) Design and Locality, 6) Design and 
Decolonization, 7) Posthuman Design and 8) Design, Data, 
Software and Artificial Intelligence (Al). Depending on what 
is decentralized in the design research and practice, 
these categories can be analyzed along eight dimensions: 
a) meta (shifting elements to the meta level), b) who 
(changing whom of the different actors have agency), c) 
what (changing what is designed), d) where (displacing 
the location of design activities), e) how (transforming 
methods and approaches), f) tools (changing the tools 
adopted), g) process (transforming processes), h) scale 
(changing scale of the design initiatives). This structure, 
displayed in Table 1, is a first approach at assessing 


the level of decentralization in design projects and 
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activities, depending on how many turns and dimensions 
are decentralized. Within such Design Decentralization 
Score (DDS), each turn is equivalent to 12.576 of a 

full decentralization; if we consider both turns and 
dimensions, each cell in the table represents instead of 


1.562576 of a complete decentralization. 


As a starting point, the turn of DIY approaches to 
Design (1) is a foundational point that challenges who 
is the designer; the boundaries between amateur and 
professional designers are increasingly blurred [22—24]. 
Following the pioneering example of Enzo Mari [25] 
makers but and designers embrace and promote this 
turn for the development of projects [26]. Moreover, 
the DIY approach has been applied by both designers 
and makers to the production of new resources and 
materials [27]. This turn also brings changes to where 
design is done (e.g. at home or in a makerspace), 

and what tools can be used in a nonprofessional 
environment, often pointing out how design practice 


can be part of the informal economy [28]. 


The turn of Open Design (2) is an extension of the first 
one, as developing projects independently is often based 
on the sharing of projects, tools and documentation, 
including the Openness, Openness and P2P turns. 

After the pioneering work of individual designers 

like Ronen Kadushin [29] and of experimental digital 
platforms like Thinkcycle [30], now Open Design is a 


Table 1. The Decentralization turns in Design: turns and dimensions 


meta who what where how tools 


DIY Design 


Open Design 


Network 


Distributed 


Manufacturing 


Locality 


Decolonization 


Posthuman 


Data, Software, Al 


process 


scale 


common practice thanks to makers and well established 


platforms like Thingiverse [31]. 


The often collaborative and collective dimension of 
Open Design brings the category of a 3) Network Science 
approach to Design, mainly for research purposes at the 
moment, with the aim of uncovering how the social 
interactions influence design processes and projects, for 
example platforms such as Github [32] or Thingiverse 
[31]. Network analysis is typically implemented in 

order to understand 1) structural dependencies within 
products and systems, 2) communication, cooperation 
and competition in design processes, 3) social, 
economic, institutional environment of design processes 
[33]. 


The often distributed nature of Open Design initiatives 
brings the increasing relevance of a 4) Distributed 
Manufacturing ecosystem of Fab Labs, Makerspaces, 
craft workshop, micro-factories and other digitally 
connected places where digital fabrication and 
traditional processes enable the manufacturing of open 
projects [4, 5, 15, 16]. 


Distributed Manufacturing systems are not only a digital 
layer that erases the local differences, but enhances 
them, as several examples of maker initiatives that 
re-evaluate local traditions and craft show [34]. Several 
examples of projects designed with and for digital 
fabrication have been collected [35], and an interesting 
case is Circular Knitic by the artists Varvara Guljajeva 
and Mar Canet, an open source digital knitting machine 
[36]. 


In this category, the Maker Movement further 
promotes the work on 5) Design and Locality of previous 
design initiatives [37, 38]. This turn focuses not only 

on users and markets for developing products and 
services, but on the local conditions, communities, 
economies and cultures and work at improving them 

at the same time. An example in this case is the Fab 
Loom [34, 39], a traditional Peruvian loom by Walter 
Gonzales Arnao redesigned and fabricated with digital 
technologies which enables collaborative processes in 


its development, usage and dissemination. 


The role of locality in design brings the importance 


of Decolonization (6) in its approaches, which further 


extends the previous turn by strongly focusing on the 
empowerment of local communities through projects 
that can develop a design approach following the local 
culture [40]. This turn is emerging in several maker 
initiatives [41] but it is however less explored, and 
further research should clarify if and when the Maker 
Movement is really advancing decolonization, or instead 
proposing a new form of technological and cultural 
colonization, or both at the same time for the moment. 
An example of this can be found in the many initiatives 
of indigenous communities for building drones for 
community mapping activities that aim at reclaiming 
ownership of their territories against government 

and companies [42]. Drones are an iconic project 

in the Maker Movement, and not only makers and 
designers are using them, but also designing, fabricating 
and releasing them openly as Noumena did with the 
NERO project [35], who then also deployed it in Costa 
Rica for the 3D scanning of pineapple plantations in 


collaboration with local makers [43]. 


Further along this direction, the Posthuman design 

turn (7) supports emergent critical race, gender and 
decolonial theories. Therefore, Posthuman in design 
changes the perspective of what is agency, who has 
agency, and how it can shape interactions. In the 
Decentralization Turn, posthumanism shifts the agency 
from the human who was placed in the center of the 
design actions, and takes into the consideration the 
locality and different agents — like technology, the 
environment and other entities as also capable of 
shaping interactions. In the Maker Movement, this 
approach often takes place with work on synthetic 
biology projects that enables living organisms to 
produce materials and artifacts; makers have already 
started coupling this with open source 3D printers 
[44, 45].The decentralization of the role of human 
actors towards non-human ones in terms of agency is 
also connected to the increasing connections among 
8) Design, Data, Software and Al, where software is 

not just a tool any longer but a creative actor in the 
generation of projects [46, 47]. While Generative 
Design approaches have been around for decades, with 
the digital fabrication technologies and the attitude 
provided by the Maker Movement designers like MHOX 
are increasingly treating software as a design material 


that acts semi-autonomously [48]. 
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4. Conclusion 


Based on a critical reflection on the literature, this 
research explores different turns in the Design research 
and practice. We propose not a new turn, but a set 

of turns called the Decentralization turns which 
represent a clear force towards decentralization along 
the integration and extension of the previous turns. 

As an initial step towards its development and better 
understanding, we propose a set of decentralization 
forces, divided by eight turns and eight dimensions. 

We use the Maker Movement as the starting point, 
context and example of the preliminary analysis of these 
forces. This proposal aims at contributing more insights, 
approaches and practices to the Design research and 
practice, highlighting emerging possibilities and their 
architecture. However, further analysis is necessary in 
order to verify and suggest new modes of operation 
regarding distributed, hyperconnected, and complex 
intelligent ecosystems. The next steps can be directed 
towards a) the exploration of their impacts on the 
design process, user experience, social consequences, 
and the role of designers, b) the validation of the 
proposed score through the developing a composite 
index based on multiple data sources and capable of 
integrating the eight turns and dimensions [49] in order 
to measure the level of decentralization. Finally, as 
consequence of the lack of consensus on the concept 
of turns and paradigms in design, there is not a strong 
theoretical background: a more rigorous analysis of 
turns and paradigms in design research and practice is 


necessary. 
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The Domestic Shape of Al: 
A Reflection on Virtual Assistants 


Abstract 


Artificial intelligence is more-or-less covertly entering 
our lives and houses. In this context, virtual assistants 
such as the renowned Amazon Alexa and the products 
that embed them, are the most representative of the 
first wave of materialization of artificial intelligence in 
the domestic domain. We focus on these products, 
studying their tangible form and appearance as well 

as the interaction modalities of 10 domestic virtual 
assistants already in the market or close to be. Our 
analysis highlights two main approaches in terms of 
shape: on the one hand Al is embedded into common 
home appliances — usually speakers — while, on the 
other hand, robot-like assistants are taking ground. 
Concurrently, we portray a still immature reflection 
on this issue from a design perspective. We highlight a 
frequently poor translation of functions into tangible 
shapes as well as a lack of reflection on interaction 
design basics, such as: input and output modalities, 
feedback systems and functions discoverability. The 
subsequent discussion frames the results in a broader 
reflection about shape, function and meaning paving 
the way to alternative paths to follow for the design of 


future home virtual assistants. 


Keywords 


Al, Interaction Design, Home Virtual Assistants, 


Tangible Forms, Interaction Modalities 


52 Design and Semantics of Form and Movement 


1. Introduction 


Aware that the world is gradually moving towards the 
Ubiquitous Computing that Weiser envisioned in 1991 [1], 
and computation is spreading throughout the physical 
space and across multiple devices to build environments 
that help people in their ordinary activities [2], we want 
to frame the transitional period we are experiencing, 
looking for a path to follow in design research and 
practice. In particular we look at those products 

we consider representative of the first wave of 
materialization of Al in the domestic landscape, namely 
virtual assistants. Today, conversational Al-based agents 
such as Amazon Alexa, are spread in 100 million houses 


according to Amazon’s SVP [3]. 


If studies have been carried out on the UX of 
conversational agents [4] and on how Machine Learning 
(ML) is/can be integrated into the UX [5, 6], the 

design discipline still has not investigated how Al and 
its skills have been materialized into objects. This 
aspect is the focus of the paper that analyses those 
products specifically designed to act as domestic virtual 


assistants. 


The aim is to initiate a discussion in the design domain 
on the tangible appearance of Al-enabled virtual 
assistants, focusing on the shape these objects have, on 


how they embody skills and interact with people. 


1.1 Al-enabled Virtual Assistants: an Analysis 


For years Al has been covertly pervading systems and 
devices that we use daily, but recently, the exigence 

to state the existence of Al technology resulted in 
specifically designed products. As a consequence, 
domestic multi-purpose assistants, materialized the 
smartness of Al in the shape of smart speakers that 
can learn through continuous conversations with their 
owners. Furthermore, in the last years, affordable 
domestic robots, has entered the market opening a 
frontier that is closer to the idea traditionally associated 
with Al: that of sentient robots able to simulate human 


behavior. 


These applications exemplify a double interpretation of 
Al, which has been animating the scientific debate for 
years: on the one hand, McCarthy's position focused 
on the creation of a super brain capable of simulating 
human behavior while, on the other hand, Engelbart's 
position based on the amplification of human potential 
through Al [7]. 


Taking this dichotomy into consideration, we analyze 
here ten devices integrating Al agents (Figure 1), the 
only in the market with the characteristics listed in the 
following. They are (i) multipurpose home assistants 
with no other distinct goal (i.e. elderly assistance, etc.), 
(ii) specifically designed as first-party hardware, (iii) 


already commercialized or coming in the near future 


E . 


Amazon Echo Google Home 


Fig. 1. Pictures of the 
analyzed domestic 


assistants (authors' graphic 


elaboration of official 


Apple HomePod Emotech Olly 


producers' images) 


and (iv) able to control other smart home appliances. In 
case of families of products, we only considered the first 


released in their latest version. 


The investigation on domestic virtual assistants starts 
from the analysis of their aesthetics of interaction 

[8], as evidence of Al embodiment. It encompasses 
physical appearance, use and interactivity, in preliminary 
considerations. The study here presented does not 

aim at evaluating the user experience of the analyzed 
products, but to portray the state of the art of domestic 


assistants, aiming at stimulating further reflections. 


Hence, neither UX assessment protocols [9] nor 
usability tests [10] have been employed to analyze 

the declared features. Products have been evaluated 
through official videos and documentation from their 
producers. A method considered suitable, given the aim 
of the research and that some of the products are still 


not in the market. 


Table 1 lists the products analyzed (arranged by 
release date of the first model) and crosses the 

basic parameters we considered for the analysis: (i) 
physical appearance, (ii) input and output modalities, 
(iii) feedback systems and (iv) discoverability [11] of 
functions, specifically considering how proactive those 


artifacts are. 


Mji Tapia Asus Zenbo Jibo 


| 0 


—— 


= 
InGen Dynamics Home Connect Samsung 
Aido Mykie Galaxy Home 
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These parameters have been chosen to analyze and 
describe, in a very basic manner, the selected products 


from a product/interaction design standpoint. 


A choice motivated by authors' main hypothesis about 
domestic virtual assistants. Acknowledging the great 
market success of some of them, this paper aims at 
showing how immature they are in some respects and 
at encouraging designers to take on a leading role in 


identifying a language and a meaning, beside a form. 


2. On the Embodiment of Virtual 
Assistants: A Review 


Physical Appearance. Domestic virtual assistants 
share quite similar appearance in terms of main 
colors and materials. They cover the grayscale with 
white being the most used color, a choice that may 


be inspired by sci-fi imaginary. On the other hand, 


Table 1. Comparative analysis of home virtual assistants 


black and grey derive from traditional hi-fi aesthetics, 
underlying their main functionality as speakers. Plastic 

is the predominant material, while fabrics are getting 
ground, seeking a better integration within the domestic 


environment. 


Shape is the most significant feature. The review 
highlights two main formal paths: on the one hand, 
there are smart objects following simple and mainly 
regular shapes; on the other, assembled bodies are built 
according to the geometric addition of solids [12], in 
four cases looking for a characterization as human/ 
animal like shape with a recognizable head and body. 
The separation of formal outcomes also highlights a 
different functional purpose, giving proof of the fact 
that the intended tasks have a relationship with the final 
shape: smart speakers are the first embodiment of Al, 
and as such a natural outcome for a speaking technology. 


Home Pod, for example, is totally focused on audio 


Amazon Google | Mji Tapia Asus Jibo Apple Emotech InGen Home Samsung 
DEVICES Echo Home Zenbo Home Pod Olly Dynamics | Connect Galaxy 
Aido Mykie Home 
Year of production 2014 2016 2016 2017 2017 2018 2018 2019 tba tba 
Simple Shape 8 © e o 
PHYSICAL Assembled solids e e ( e e e 
APPEARANCE 
Main Color & Material B/G, F W+G, P W, P W, P W+G, P W/B, P B,P W, P W, P B, P 
Proactive e e e 
DISCOVERABILITY 
Non Proactive (J e e J e e e 
Voice e e [J o e e ® e LJ e 
App e e e o e oO e (J e o 
Buttons e e e e e e e e 
INPUT Touch Surface e [s] e 
Touch Display e e e e e 
Vision (camera) e e e e e 
Touch e o 
Other Device Action e e (J e e e e e e o 
Audio e o e e e e e e e e 
OUTPUT 
Video e e e e e 
Movement e e e e e o 
Lights e e e e [) e e 
Voice e e e e e ° e e e o 
FEEDBACK 
Movement e e e 
Display e e e. e e 
Colors: B-black; G-gray; W-white. Materials: F=fabric; P=plastic. 
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quality and it looks like nothing but a speaker. On the 
contrary, devices aiming at establishing a social contact 
assume a more anthropomorphic shape. For instance, 
Aido is thought of as a butler and its height and shape 
nurture this concept. So, functions highly influence the 
general, formal configuration of the object, also in terms 
of dimensions. Speaker-shaped assistants are nothing 
more than discreet ornaments, while those with social 
qualities increase to the dimensions of small home 
appliances or become bigger, in case of a closer human 


simulation. 


The possibility of movement may contribute as well to 
the embodiment of function: Olly has neither a simple 
shape nor an anthropomorphic one, but its movements 


nurture a sense of social connection. 


Behavior and Discoverability. Al assistants are 
based on ML, then evolving according to their owners' 
preferences is a common feature. Again, what influences 
the behavior of domestic virtual assistants is their main 
purpose, once more underlining the distinction between 


smart speakers and domestic robots. 


Most of the assistants, namely smart speakers (Amazon 
Echo, Google Home, Apple HomePod, Samsung Galaxy 
Home) and, surprisingly, some robot-like assistants 
(Zenbo, Jibo and Aido) have a non-proactive behavior: 
they are unobtrusive and respond only when prompted. 
On the contrary, three assistants are also proactive and 
suggest information, activities or contents to their users 
according to their habits, mood or expected necessities. 
To better perform their proactivity, they integrate a 
camera (or more), so that they can relate on more data 
to foster their suggestions: they not only evaluate noises 
or routines, but they also read body language and can 
understand what their users are doing. Furthermore, 
they can recognize and be triggered just when their 
users are passing by. The highest point in terms of 
empathic interaction and proactivity is represented by 
Olly, which develops and manifests its own personality 
according to its interlocutor's one. The discoverability 
of non-proactive objects is really low, and most of their 
functions remain obscure to the user. This is the case 

of Alexa: it has thousands of skills, mostly created by 
third-party developers, but rarely known and used [13]. 
Interaction. The investigation on interactivity has 


been limited to input, output, and feedback modalities, 


referring to Saffer's Systems Design [11]. The main inputs 
and outputs are vocal, highlighting one of the most 
important achievements of Al towards a more human 
interaction. Indeed, Natural Language Processing (NLP) 
has improved to the point that it can easily understand 
human requests and answer accordingly. This is the 
main premise for the rising of digital assistants, which 
represent the best evidence of this technological 


accomplishment. 


Another feature that all of the devices still share as 

an input is a through-app interaction: its functionality 
ranges from basic setup to complete functions 
(especially for all the smart speakers). Moreover, they 
also have buttons to perform specific activities: a mute- 
microphone button and, respectively, volume up/down 
buttons or touch surface. On the contrary, robot-like 
assistants only need the starting one as they have a 
touch display as a face. Additionally, proactive devices 
take advantage of their cameras to read body language 
and gestures as inputs. While, finally, some of them 
(namely Jibo and Aido) emotionally respond to touch 


inputs as a result of being cuddled. 


In terms of outputs, all the assistants allow the 
interaction with other home appliances, and the 
providing of audio contents: web researches, music, 
podcasts, etc. For the devices having a display or a 
projector (Aido and Mykie), video contents are also an 
output, which can be a reproduction of internet sources 
or an enriching characterization of what they are saying 


or doing. 


Movement is another possible output: it can be the 
result of a request — dancing (Jibo) or moving across the 
rooms (Aido and Zenbo) - or just a reinforcement for 
communication — moving up and down while counting 


push-ups (Olly). 


What is more relevant in an interpersonal-simulated 
interaction, though, is the feedback system. Relying 

on [14]’s framework to analyze human-product 
interaction, it results that Al-enabled assistants offer 
almost no inherent feedback, as physical actions are 
required in a very limited manner; then only functional 
feedbacks (corresponding to the described outputs) and 
augmented feedbacks characterize the current domestic 


assistants. In particular, lights, verbal utterances, 
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movements and displays reveal the internal state of the 
object while the function is processing. Almost all of 
the objects use lighting systems to show their current 
status, especially the speaker-based assistants, whose 
choreographies remind those they already have on 
other devices (for example, it is the case of Google 
Home bouncing dots). It is to be noticed that — except 
for Galaxy Home - all the brands have opted for 
colorful lights, that become particularly expressive in 
the Olly's custom-built circular LED display: it really 
underlines the effort of creating a patent communication 


system through lights. 


Another typical feedback of digital assistants comes 
from their voice: whether in a rigorous or more 
confidential way, with a robotic or person-like tone, 
these devices let their users know if and what they 
have understood before they provide the requested 
content. This kind of feedback is especially positive in 
the interaction with anthropomorphic assistants as 

it gives the impression of being engaged in an actual 
conversation with a companion, and not just being 


talking to a machine. 


For the devices having a display and being able to move, 
those are great occasions for feedback: natural and 

fluid movements can follow the activity or stress the 
awareness of the robot in its being addressed to its 
user — for instance turning its head towards the one 
who is speaking. Displays, instead, are used to show the 
bot's more or less abstracted eyes, which animate in 
relation to the user's inputs. Those feedbacks are the 
most effective in providing a natural interaction, as the 
machine appears more alive, expressing its own identity. 
An aspect that is strongly related to feedback, and 
highly influences the interaction, is feedforward [14]. 

In the example of virtual assistants, the user (and 
potentially the designer and/or programmer) does 

not know what will happen after making a request. In 
many cases (s)he can only imagine or expect a certain 
outcome, while one can be absolutely sure of the more 
usual interactions (through app or buttons) and of the 
basic and routine commands that (s)he has performed 
various times. Otherwise, the output of the interaction 
is unpredictable and — additionally — not immediate: two 
aspects that may hinder the perceived quality of the 


interaction. 
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3. Reflections on the Embodiment of AI- 
Enabled Assistants 


The discussion of the formal outcomes cannot be 
separated from the main functions of the object. 

As remarked, the shape derives from use and from 

the relationship that the devices have with their 

context. Except from its inclusion in the name of some 
products, the concept of home has been poorly taken 

in consideration while designing the appearance of 
those objects. All of them are devices that can be freely 
set in any environment, and still have not created a 
connection with our domestic reality: a condition which 
may contribute to make people perceive them as aliens 
in their houses. Perhaps, a closer relationship could help 
users to understand the assistant's benefits more easily. 
In the meantime, they are interpreted according to their 
similarity to other, known, objects, or to the abstract 
expectations that the speculations of our culture have 
encouraged. Then, it does not come by chance that 

the most common use of speaker-shaped intelligent 
assistants is to play music [4], despite the introduction 
of NLP features opened novel opportunities. On 

the other hand, devices with a humanoid figure are 


commonly defined as companions or home managers. 


Then, coming to their functionalities, it emerges 

that users prefer few known commands that actually 
transform the intelligent assistants into mere executors 
of routines, especially talking about smart speakers. 
Hence, the multiple skills they have are rarely used, 

as discussed by [15] and confirmed by [4] through 
quantitative and qualitative analysis of use of Alexa. The 
study by [13] underlines that discoverability in smart 
speakers is a big issue for designers and it proposes 
possible solutions to overcome the problem, including 
context awareness and proactivity. This condition of 
poor discoverability and weak affordance [16, 17] may 
be due to a deficiency in the embodiment of the skills 
into tangible products, which often mark a rejection 

of sensuous curiosity and pleasure [18—20]. In fact, 
when executing actions, users are prevented from 
experiencing their inherent effect. In particular, [14]'s 
framework may be employed to pursue a more intuitive 
interaction. In order to strengthen its quality, the 
authors suggest restoring natural couplings between 
actions and reactions according to six different factors. 


At the moment, the most promising and easy-to- 


integrate solutions are related to the expression of 
interaction: the output modality could be a reflection 

of the conditions in which the request is performed, 
such as the user's mood or the moment of the day [21]. 
Currently just three of the analyzed devices (Olly, Mykie 
and Tapia) feature these functionalities, but it could give 
true value to the overall interaction and utility of the 


assistants and, possibly, inform their shape. 


One hypothesis is that a humanized aspect and 
behavior can make an Al-enabled device truly appear 
like a domestic assistant. Probably its association 
with a human being sets the conditions for a more 
natural interaction and makes it simple to imagine 
that the object has covered skills and it is likely to 
have a proactive role in our daily life. Yet, we are 
incline to anthropomorphize everything [21], and 
functions — made explicit through proactivity —, shape 
and movements are what actually seems to facilitate 
users in terms of discoverability and, therefore, 
interaction. Furthermore, the behavior of those devices 


characterizes them with well-defined identities. 


In the light of this reasoning, another consideration 

can be pointed out: a distinctive trait of some 
anthropomorphic assistants is what defines them as 
mutuality of influences systems [18], meaning that they 
are sensitive to perceptual crossing. It is not only the user 
who has to perceive the object of interaction, but it 
also has to sense the individual who is going to trigger 
it, preparing and giving notice of this awareness. In this 
way the interaction becomes expressive, embodied and 


responsive, even without the use of an interface. 


Deeper connections between shape, abilities and 
behaviors should be developed according to a more 
mutual relationship between objects and people, trying 


to understand what it is really about. 


Facing the complexity of these devices and their role 
in the ordinary life, another issue designers must 
reflect on is the materialization of a great amount 

of skills, especially if the only means of interaction is 
conversation. In fact, most of the analyzed assistants 
feature a companion app to be accessed through 
smartphone, allowing the personalization of the 


product as well as a traditional browsing of the skills. 


Yet it contradicts one of the premises for a natural 
interaction: getting rid of interfaces to make the 
mechanism of the artifact directly available to the 


system users [22]. 


Undertaking the same direction, a similar, and perhaps 
less intuitive, interaction is allowed by the devices 
integrating a tablet as head, which can be actively used 
for input and output. An arguable solution that entails 
an evidently poor investigation about the meaning of 


interaction. 


Analogously, the use of voice as main tangible 
manifestation of the intelligence behind domestic 
assistants does not automatically make them real 
conversational agents, nor it makes it easy to exploit 
their potential. This condition is made evident by the 
choice of every producer to integrate added input 
systems ranging from simple buttons to cameras and 
complex nested menus to be browsed through bespoke 
apps. In other words, the conversational agents are 
frequently thought of as the tip of an iceberg that 
should make interaction more human and friendlier but, 


actually, they are still in need of a better definition. 


4. Future Design Scenarios for Virtual 
Assistants 


The first wave of materialization of Al into the domestic 
domain, as described in the introduction, is still 
profoundly immature in terms of function, language and 


meaning [23]. 


Designers are called to manage the coupling of form and 
ever-increasing functionalities, finding a compromise 
between expressing them or keeping them implicit. The 
current situation still mirrors the dichotomy animating 
the discourse within the field of Al. On the one hand, 
products like smart speakers materialize Al into 
aesthetically appealing objects inhabiting our houses. 
On the other, objects with varied shapes aim at being 
perceived as valuable humanoids to help in ordinary 
life. A dichotomy that permeates all the aspects — with 
regard to physical appearance, behavior and interaction 
— analyzed in the study and betrays a still immature 
reflection from a design perspective, and the need to 


find an original language. 
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Even though the argumentation only considers 

few initial parameters and presents limitations 

better addressed in the following, some preliminary 
considerations may arise. The impression is that the 
field is still experiencing that period of drunkenness that 
characterizes the introduction of every new technology 
[24]. Domestic assistants seem to be in the toy phase 
having to find their actual contact with reality [25], and 
design should take a leading role in guiding a human- 
centered transition towards meaningful products. 
Present perspectives lead to the integration of Al 
discourse into the Interaction Design field. In particular, 
they share the goal of fostering a more natural 


interaction, going beyond display-mediated interfaces. 


Then, translating Al in a tangible and domestic form, 
according to a human-centered and holistic perspective, 
shape cannot be separated from function and meaning. 
From what seen, Al-enabled objects could simplify 
their functional structure so that, from a tangible point 
of view, the interaction could be more immediate and 
significant in terms of experience and utility. In this 
sense, we could be facing three main scenarios: (i) 
virtual assistants could evolve as self-standing objects, 
maybe with more specific and limited functions that 
better translate into a clearly recognizable form; (ii) 
they could become more accurate humanized robots, 
taking the role of actual people at service of others; or 
maybe, (iii) they could be completely dematerialized 
and spread across other existing appliances and 
devices throughout the physical environment, with a 
sensibility for their location and proximity to each- 
other, accomplishing the ideal of Ubiquitous Computing 
[22]. The latter, in turn, may open further enquiries 
about the embodiment, regarding its scale (product or 
environment) and the way the augmented functionality 


will be expressed. 


Clearly, the study presents several limitations. It takes 
into account a small sample of products, even if they 
represent the totality of those responding to the 
selection criteria. A broader study may include the 
entire range of products of the producers here analyzed 
and, eventually, third-party hardware integrating Al 
agents. Furthermore this initial argumentation, starts 
from the fundamental traits of a product, but may 
expand triggering a discussion about current physical 


materialization of Al that may comprehend other 
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design matters like UX, or interdisciplinary reflections 
like social/psychological implications, as the emotional 
response they may nurture through interaction, as well 


as experimental studies. 
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Conversational Smart Products: 
A Research Opportunity, First 
Investigation And Definition 


Abstract 


This paper approaches the topics of Conversational 
Design and Conversational Interfaces (CUI) from 

a product design perspective. While this topic is 

more widespread in HCl and UX design community, 
there is a lack of insight coming from product design. 
Given that the integration of CUls and assistants 

into physical products have the potential to impact 
product design practice, the paper introduces the term 
Conversational Smart Products (ConvSP). It analyzes 
the topic of Conversational Design and the construct 
of “Product Smartness” and presents a framework 

to define and analyze ConvSP. The research aims to 
delve into ConvSP characteristics, deepening how the 
conversation can be embodied in the product, and 
how the physical attributes could work in synergy with 
the verbal dimension. Through a case study research 
conducted on a selection of 30 existing products with 
CUls, insight is collected on four different topics. On 
product smartness and personality; on the level of 
physical embodiment of the conversation; on request/ 
response design; and on the tangibility of conversational 
inputs and outputs. A descriptive definition of 
Conversational Smart Products is given. Broadly, the 
research aims to investigate what the role of product 


designers in the development of ConvSP could be. 


Keywords 


Conversational Interfaces, Smart Products, 


Conversational UX, Product Design, HCI 
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1. Introduction 


The idea of building machines capable of behaving and 
dialoguing as human beings has influenced Artificial 
Intelligence (Al) since its emergence as a field of 
research. Since the release of the vocal assistant Siri in 
2011, Natural Language Interfaces (NLI) and assistants 
have become a focal point in both academic and 
industry research. There are several Conversational 
User Interfaces (CUI) able to mimic human, turn-based 
conversation and to use natural language in written or 
spoken form [1]. Their aim is to provide for a more 


natural and immediate interaction. 


CUls have become increasingly widespread and 
integrated into physical smart products. This will have 
an impact on the practice of product design, especially 
for consumer electronics. The topic of conversational 
design is gaining importance within the Human 
Computer Interaction (HCI) community [2] and from 
the point of view of User Experience (UX) Design. In 
particular, there is a drive to formalize “Conversational 
UX Design" as a distinct discipline in UX Design [3-4]. 
There is a current lack of research on this phenomenon 
from the perspective of Product Design. Therefore, 
the paper introduces the term “Conversational 

Smart Products” (ConvSP), and it delves into their 
characteristics. It discusses the results of an analysis 
conducted on 30 existing products with conversational 
interfaces. It offers product design insight and presents 


a working definition of ConvSP. 


This paper is part of ongoing research aimed at 
understanding what could be the role of product 
designers in the development of conversational smart 
products. It gives particular attention to how products 
embed CUls and how the physical attributes could 
work in synergy with the verbal dimension during the 


interaction. 


The structure of the paper is the following. Section 

2 describes the meaning of the term Conversational, 
illustrates the different conversational interfaces, and 
outlines how Design is approaching the topic. Section 

3 deepens smart products and their characteristics. 
Section 4 presents the research topic of Conversational 


Smart Products. Section 5 explains the case study 


research carried out on 30 existing products with CUls. 


Section 6 discusses the results, and emphasizes four 


main topics. 


2. Conversational Interfaces and 
Conversational UX 


“Conversational Al” is a term used in Artificial 
Intelligence to represent the study of techniques aimed 
to create software agents that can engage in natural 
conversational interactions with humans [5]. The term 
is historically used applied to interfaces to describe 
systems that display human-like characteristics and 
support the use of spontaneous natural language in the 


form of text or voice. 


Conversational interfaces include [1]: 


Chatbots. Chatbots are computer programs that 
process a natural-language input of the user in text 
form and generate a textual response to the user. 
Their history goes back to the 1966 with ELIZA, the 
first chatbot developed at MIT by Joseph Weizenbaum. 
Chatbots can include visual elements and rich 
interactions [6]. Chatbots are currently being tested 
for forms of “Conversational commerce", brand 


engagement, and integrated in messaging platforms [7]. 


Vocal interfaces (VUI) are based on a similar 
technology but have the added complexity of dealing 
with speech inputs and outputs [8]. 


Virtual Personal Assistants (VPA) - also called 


personal butlers or Al assistants - are programs 


integrated in a device, operating system or app, that 
can take over a multiplicity of tasks for the users. 
Assistants are "super bots" that facilitate and manage 
multiple services [6]. They have a personality and a 


character, like Amazon's Alexa and Apple's Siri. 


Embodied Conversational Agents (ECA) are 
agents in form of animated characters on screens. Their 
aim is to simulate face-to-face interaction with human- 


like agents that use their bodies in conversation [9]. 


"Conversational User Interfaces" (CUI) is one of the 
terms used to identify Natural Language Interfaces. 
Cathy Pearl, Head of Conversation Design Outreach at 
Google [8], states that the term Conversational Design 
should only be used to identify those systems in which 
interaction can go beyond one turn. This because 
human conversation is bidirectional and turn-based. 
During the conversation, each participant can take the 
initiative, exchange information, and keep memories of 
the past turns of the interaction. Machines instead are 
not always able to keep this memory and go beyond 
simple one-shot requests, therefore simulating a 
conversation. For the same reason, Porcheron et al. 
[10] propose the term "request/response" design to 


describe interaction with VUls. 


Hall [11] instead uses the term Conversational with 

a broader perspective. According to her: "taking a 
conversational approach to interaction design requires 
applying the deeper principles of how humans interact 
with one another". A truly conversational system should 
manifest conversational qualities at a deeper level, not 


only as a facade. 


This research uses the term Conversational because it 
does not focus on the linguistic capabilities of the agent 
but refers to the emerging discipline of Conversational 
UX Design. User Experience (UX) is an emergent 
discipline that focuses on encompassing all aspects of 
the end user interaction with companies, their services 
and their products. It is a multidisciplinary field that 
considers the entire user journey. Usability and User 
Interface (UI) design are merely subsets of this concept 
[12]. 


Conversational UX design means being able to 


create experiences that work like a conversation. 
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Moore et al. [3-4] consider it necessary to define it 

as a distinct discipline in UX Design. This is because 
human conversation is a complex system that requires 

a different skill set/background compared to the 
development of other interfaces. Designers must 
become conversation experts and draw knowledge from 
different disciplines that study human conversation such 
as Discourse Analysis, Interactional Sociolinguistics, 
Conversational Analysis, Automatic Speech Recognition 


(ASR) and Natural Language Processing research (NLP). 


The constant increase in products’ processing 
capabilities and connectivity facilitates the diffu-sion 
of smart products with conversational interfaces. For 
this reason, Conversational UX De-sign should include 


contributions from a Product Design perspective. 


3. Smart Products and Product Smartness 


Smart products are a product category that represents 
internet-connected consumer electronics. Other terms 
such as intelligent products and connected products 
have been used to describe the same concept. These 
objects are reshaping industry boundaries and creating 
entirely new industries, raising a new set of strategic 
choices and disrupting value chains [13]. Then, what 


makes a product smart? 


Rijsdijk and Hultink [14] define product smartness as 

a construct consisting of seven dimensions: autonomy, 
adaptability, reactivity, multifunctionality, ability to 
cooperate, human-like interaction, and personality. The 
level of smartness of a product depends on the extent 
of its capabilities in one or more of these dimensions. 
However, these functionalities must be obtained 
through the use of Information Technologies (IT) for 
the product to be described as "smart". Although a 
universally accepted definition of “smart product” does 
not exist, the term identifies objects that share some 
common technical characteristics. Different authors 


propose definitions and technical requirements [15-19]. 
Smart products share three main characteristics. They 
are cyber-physical, networked, and with computational 


intelligence. 


Cyber Physical [19]. Smart products blend hardware 


and software. They are physical objects with a digital 
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representation. Interaction is a key aspect, and it can 
occur through multiple, multimodal interfaces. For 
example, there could be a direct interaction with the 


product combined with an external web application. 


Networked. Smart products are networked and 
uniquely identified. Internet connection makes them 
part of a larger network of things, people and services. 
They can communicate with users and/or with other 
objects. Connectivity can take three forms, that can be 
present at the same time: one-to-one connection, one- 
to-many, many-to-many [12]. These connected products 
are part of the so-called “Internet of Things” [20] and 


are enhanced by a strong service component [21]. 


With Computational Intelligence. In contrast to 
the passive nature of most products, smart products 
display autonomous and proactive behaviors [15-16]. 
This blurs the boundaries between product design 

and robotics, and it leads to reconsider the notion 

of product agency - that is the ability to act and 
produce effects according to an intention [22-23]. 
Computational intelligence may be located in the object 
(edge intelligence) or even completely outside the 
physical products, e.g. in the cloud [18]. Computational 
intelligence enables products to be context-aware, 
personalized, adaptive and anticipatory. These four 
characteristics derive from the literature on Ambient 
Intelligence (Aml), a technological vision developed in 
the late nineties that envisioned environments infused 
with technology, sensitive and responsive to the 
presence of people [24]. Smart products were seen as 


the building blocks of Ambient Intelligence [17]. 


The four aspects of computational intelligence according 


to Aml are briefly summarized here. 


Awareness. In products, awareness can occur in 
several forms. User-Awareness means that the system 
is able to recognize and memorize users and their 
preferences. Contextual-awareness refers to the ability 
to detect what happens in the environment and to 

infer user intentions according to the situation. Self- 
awareness means that the product has and generates 
new information and memories about itself, its use and 
purpose. Cultural awareness means that the product is 
socially appropriate. For example, a talking toy that uses 


a language appropriate to the children's age. 


Reactive, Adaptive, Anticipatory. Reactiveness is 
the ability to quickly react to events and environments. 
Adaptability is the ability to use data to respond and 
adapt behavior over time, even according to user needs 
and desires. An anticipatory product can proactively 
anticipate users' plans and intentions [16], and suggest 
actions. 

Thanks to these features, smart products offer 
opportunities for envisioning new functionalities 

and interaction. They raise new challenges for their 
design, such as designing interactions distributed 
across multiple devices, in which the focus of the user 


experience is the service and not the product itself [25]. 


4. Research Perspective: Conversational 
Smart Products (ConvSP) 


Smart products with conversational interfaces and 
virtual assistants are becoming widespread. McTear, 
Callejas & Griol [1] explain that the rising interest 

in conversational interfaces depends on five main 
factors: (1) the renaissance of Al as a field and 
advances in language technologies; (2) the emergence 
of the semantic web; (3) the improvement in device 
technologies; (4) increased access to connectivity; 
(5) the interest of major companies. In addition, 
platforms and libraries [5] support the phases of design, 
prototyping, and development of CUls. For example, 
the design software Adobe XD added in 2018 the 
possibility to simulate voice interactions in interface 


prototypes. 


CUls are a topic of interest for Human Computer 
Interaction (HCI). In 2018, Clark et al. [2] did a 
systematic review on the status of speech interfaces 

in HCI. The study found that current research in HCI 
converges on nine topics. These include studies on user/ 
system speech production, comparisons on different 
modalities of interaction, investigations on people's 
experiences with personal assistants, and studies aimed 
at generating design insight. Their research highlights 
the lack of design related work and research, outside 


usability testing and theory-based research. 


This research addresses the topic of CUIs from a 
product perspective. In particular, it has three main 


objectives. 


O1: it aims to define "Conversational Smart Products" 
(ConvSP) and their characteristics; 

O2: it aims to deepen how conversation can be 
embodied in products, and how the physical attributes 
could work in synergy with the verbal dimension during 
the interaction; 

O3: broadly, it wants to investigate what could be 

the role of product designers in the development of 


Conversational Smart Products. 


Literature from the disciplines of Human-Computer 
Interaction, Artificial Intelligence, and User Experience 
Design contributed to investigate the theme of ConvSP. 
In HCI, it was given relevance to the subject of speech 
interaction and Tangible Interaction. Tangible Interaction 
is an umbrella term denoting systems that rely on 
embodied interaction, tangible manipulation, physical 
representation of data, and embeddedness in real space 
[26]. 

Tangible interaction is a promising field of research 

for smart connected products. The study of Tangible 
Interaction applied to the Internet of Things has been 
called “Internet of Tangible Things" or loTT [27], and is 
seen as a way to facilitate the control and understanding 


of loT objects. 


Design researchers are already studying the use of 
sensory language to communicate information and 
engage users during user-product interaction [28]. One 
of the research hypotheses of this paper is that non- 
verbal, tangible elements could work in synergy with the 
verbal dimension, and help users to overcome some of 
the limits of voice-interactions. For example, one limit 
is that voice is not always the most natural interaction 
option, or the most socially appropriate. Another 

issue of VUls is that users may not know or remember 
what all the tasks that the system can perform are, 
because those are not visualized on a display. That is 
why chatbots and virtual Al assistants rely on menus 
and visual elements, like buttons and cards, and smart 


speakers integrate screens. 


5. Analysis Framework and Case Studies 
Selection 


As conversational products are getting commercially 
widespread, the first step of this research consisted in a 


preliminary analysis based on case studies. Exploratory 
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case studies are an appropriate method to inquire 
little known, poorly understood issues, and whenever 
the research subject is still emerging. The aim of the 
investigation was to define ConvSP and gain possible 


insight for further research. 


A framework of five main characteristics was outlined 
to analyze existing products. The framework is the 


following (see Table 1). 


Vocal User Interface 


(VUI), Chatbot, Embodied 


Type of Conversational 
Interface (CUI) 


Virtual Personal Assistant 


(VPA) 


Conversational Agent (ECA), 


Remote, Environmental, 


Embedded 


Level of physical 


embodiment 


Type of conversation User or System initiative, 
Task-led or Topic-led 
conversation. 
Conversation goes beyond 


1 turn 


Input and Output Text, Speech, Tangible 


modalities interaction 


Product smartness Autonomy, Reactivity, 
Adaptability, Multi- 
functionality, Cooperation, 


Awareness, Personality 


Table 1. Framework for the analysis and definition of 


Conversational Smart Products 


The Type of Conversational Interface (CUI) 


includes VUIs, Chatbots, ECAs and VPAs. This criterion 


aims to identify which are the most frequent, and to 
get insight on how Assistants (VPAs) are integrated in 


products. 


Level of Physical Embodiment of the CUI. 
Fishkin's taxonomy [29] was the starting point to 
determine where and how conversation takes place 


in the product. Fishkin proposes a frame-work to 
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analyze the tangible interfaces according to their level 
of physical embodiment. He distinguishes between 
four levels of embodiment. "Full" embodiment occurs 
whenever the output device coincides with the 

input device. "Nearby" embodiment is characterized 
by the output taking place near the input object. 
"Environmental" embodiment consists of the output 
being around the user and non-graspable, like sound. 
Embodiment is "Distant" if the output happens 
remotely, such as on another screen or in another 
room. Based on this framework, three different 
levels of embodiment for conversation were defined. 
Conversation is "Remote" when the input or output 
happens elsewhere. It is "Environmental" when it 
uses sound and speech. It is "Embedded" when the 
conversation includes aspects of tangible interaction 


with the product. 


Type of Conversation. Conversation was analyzed 
in respect to three aspects. The first one is initiative. 
User-initiative means that the user can do one-shot 
queries and open-ended prompts, while System- 
initiative means that the system directs the dialogue 
with slot-filling prompts [1]. The second aspect is the 
purpose of the conversation. A Task-led conversation 
aims to accom-plish a task, while a Topic-led one aims 
to discuss and exchange ideas [6]. Finally, there was an 
attempt to check if the conversation can go beyond a 


turn of interaction [8]. 


The input and output modalities of the 
interaction of user and system were deepened to 
to see if they were only textual and verbal, or if they 


included tangible interaction. 


Lastly, the construct of product smartness was 
applied to assess whether these products possess the 
characteristics identified in the literature. Selected 
product smartness criteria integrate Rijsdijk and 
Hultink's categories and the principles of Ambient 
Intelligence described in section 3 of this paper. The 
seven criteria are: Autonomy (independent action 
and proactiveness), Reactivity (direct-response to 
environment and events), Adaptability (over time), 
Multifunctionality, Cooperation with other devices, 
Awareness (contextual, of users, of self, of cultural 


principles), and Personality (having a credible character). 


5.1 Selection Criteria and Data Collection 


The investigation was carried out on a sample of 30 
products with conversational interfaces, or compatible 
with voice assistants, launched on the market in the 
last 3-4 years. One requirement was that there would 
have to be sufficient online material to evaluate their 
interaction through video and reviews. For this reason, 
concepts and research projects were discarded. The 
product selection comes from the most recent tech 
fairs (i.e. CES 2019, 2018), official lists of products with 
built-in and compatible assistants (i.e. Alexa, Google 
Assistant, Siri, Bixby), tech blogs and blogs specialized 
on conversational interfaces (i.e. Chatbots Magazine, 


Voicebot, Wired, Tech Insider, etc.). 


The 30 case studies belonged to different product 
categories. They included: a TV remote, camera, door 
lock, earphones, food scale, commercial robot, light 
bulb, microwave, musical keyboard, printer, smart 
clock, fridge, smart outlet, shower, smoke alarm, 
thermostat, vacuum cleaning robot, washing machine, 2 
light switches, 2 smart toys, 3 smart displays, 6 smart 


speakers. 
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SMARTNESS 
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LEVEL OF EMBODIEMENT 
OF THE CUI 
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6. Results and Discussion: Conversational 
Smart Products Characteristics and 
Definition 


Products were analyzed according to the framework 
described in section 5. The alluvial diagram in Fig. 1 
summarizes the case study research in numbers, and 
highlights the relationships between the different 
aspects of the framework. 


Research findings are summarized in four main topics. 


On Product Smartness and Personality. The 
construct of product smartness proved useful to 
evaluate smart products. All the analyzed products 
possessed one or more aspects of smartness. Most of 
the products (18 p.) possessed three to five aspects at 
the same time, eight products possessed just one or 


two, while four possessed six or seven. 


Smart products are networked, and through 
connectivity enable different modalities of interaction. 
Therefore, Cooperation is the most frequent aspect of 
product smartness (28 out of 30 products), followed by 
Multifunctionality (23 p.) Awareness & Adaptiveness (18 
p.), Autonomy (15 p.), Reactivity (11 p.) and Personality 


(6 p.). 


AWARE (18) AUTONOMOUS (18) 
REACTIVE (15) 
ADAPTIVE (11) 


PERSONALITY (6) 


EMBEDDED (11) 


CHATBOT(1) ECA (1) 
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The aspect of personality, while relevant in the design of 
digital conversational agents and interfaces, it is scarcely 
present in commercial smart products. It is more 
frequent for such products to attempt to engage users 
in topic-led conversations, as done by robots, smart 
toys, and smart speakers with specific purposes, like 
teaching languages and playing with children. Most of the 
examined products integrate existing voice assistants 
(i.e. Alexa, Google Assistant) and do not create another 


proprietary character. 


Level of Physical Embodiment of the 
Conversation. Conversation happened mainly in 

an environmental, intangible way (18 products), most 
frequently through the use of voice inter-faces (23 p.). 

In 11 products, conversation took place remotely, 
according to two types of situations. In the first one, the 
input or the output of the conversation was given to or 
from another device. For example, a smart TV remote 
acted as the speech input, but the output in the form of 
voice, text, and actions took place on the TV screen. 

In the second situation, products were simply "VPA 
assistant-enabled". The products did not integrate 
CUls. External assistants, in the form of smart speakers 
of apps, received and processed the commands, while 
the product only performed actions. Conversation 

had no impact on the shape of those products. This 
characterized smart locks, smart outlets, printers, 
vacuum cleaners, showers, and smart light bulbs. 

In 11 cases, the products directly integrated the 
conversation with some tangible elements. Among 
those, four products used embedded screens to support 
the conversation. In two cases of robots and smart toys, 
the conversation was in fact "embodied" by the product 


through a character. 


Voice interfaces were the most frequent CUI (23 p.). 
Only one product used a chatbot: a washing machine 
with a chatbot assistant on a dedicated app. Only one 
smart speaker used an Embedded Conversational 
Agent: an animated cat, displayed on a screen mounted 
on top of the device. 

Since smart products increasingly embed screens, it is 
possible to integrate other kinds of CUIs beyond VUls. 
Graphical User Interfaces (GUI) could use the metaphor 
of conversation in their interactions. This could be 
useful for those products that guide users in complex 


procedures. 
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Conversation Insight: Request/Response Design. 
It was challenging to clearly evaluate the details of the 
conversation through videos and indirect references. 
Initiative was left to the user in the majority of 
samples (29 products), while it was directed in the 
case of a washing machine controllable via chatbot. 
Interactions were short and in form of commands. As 
stated by Porcheron et al. [10], this can be referred 

to as "request/response design", since there is no real 
conversation. 

Without practical tests, it was not possible to 

clearly define if the products were able to go beyond 
the first turn of interaction, as in the definition of 
Conversational Design given by Pearl [8]. It depends 
on the capabilities of the conversational agent. The 
communication capabilities of those products that have 
built-in assistants will acquire more conversational 
skills as the assistants are further developed. On the 
contrary, it is hard to define as Conversational those 
products are compatible with assistants but do not have 
CUls themselves. This has been taken into account in 
the definition of ConvSP. 


On Tangibility and Conversational Input & 
Outputs. The research shows that there are a few 
aspects of the conversation that are embodied in the 
product. The first is the "invocation" of the agent, the 
way in which users can "wake up" the product and start 
interacting with it. In five products out of 30, the voice 
interaction could only be initiated by pressing a physical 
button. This is also seen as a privacy measure, to ensure 
that products with VUls are not always "listening". In 
seven cases, screens were used to support and enrich 
the conversation. This is particularly true for smart 
speakers, that are progressively turning into smart 
displays. This opens up possibilities for multimodal 
interactions that blend voice interaction and Graphical 


User Interfaces. 


During the conversation, feedback in the form of 
animated light effects was used to communicate with 
the user that the agent was listening and producing 

a response (in eight products). There could be the 
opportunity to design multimodal feedback that 
synchronizes to the content of the conversation. For 
example, the feedback could change in case of greetings, 
confirmations, and errors. Another opportunity is to 


design ways to aid users in remembering and visualizing 


the available actions and commands that the product 


supports. 


6.1 Working Definition of Conversational Smart 
Products (ConvSP) 


Given the findings of this first investigation into existing 
products, the working definition of the ConvSP is as 
follows: 

ConvSP are physical, networked products, augmented 
by a digital counterpart. 

They possess one of more aspects of product smartness 
in the form of Autonomy, Reactivity, Adaptability, 
Multifunctionality, Cooperation, Awareness, Personality. 
They embed Conversational User Interfaces (CUI) in 
the form of Vocal User Interfaces (VUI), Embodied 
Conversational Agents (ECA), Virtual Personal 
Assistants (VPA), and chatbots. 


There are three levels of physical embodiment of the 
CUI. It is “Remote” when the input or output happens 
elsewhere, "Environmental" when sound and speech 
are used, and "Embedded" when there the conversation 
includes tangible interaction with the product. A 
ConvSP must clearly display or suggest through its 
physical form and feedback, that is conversationally 


enabled. 


7. Conclusions 


This paper approaches the topic of Conversational 
Design and Conversational Interfaces (CUI) from a 
product design perspective. It introduces the term 
Conversational Smart Products (ConvSP) to describe 
those physical products that embed conversational 


interfaces, and offers a descriptive definition. 


Broadly, this work is part of a research that aims to 
investigate what could be the role of product designers 
in the development of conversational smart products. 
The paper proposes a framework for the analysis 

and definition of Conversational Smart Products 
which considers the type of CUI, the level of physical 
embodiment of the conversation, the type of 
conversation, the input and output modalities, and 

the overall product smartness. It begins to investigate 
how the conversation can be embodied in the product, 


through a case study research conducted on a selection 


of 30 existing products with CUls. A limit to the 
present study is that it only investigates smart products 
using indirect resources such as videos and reviews. 
Therefore, the evaluation of the interaction may not be 


complete or completely correct. 


It is a preliminary study that proposes four different 
topics that can be relevant in the design of ConvSP. 

The first point is that ConvSP possess one or more 
aspects of product smartness. The aspect of personality 
is more relevant for those products that engage in 
topic-led conversations, such as robots and smart toys. 
The second point is that interaction with ConvSP is 
mostly environmental, and happens via VUI. Since smart 
products embed screens, there is space to integrate 
other kinds of CUlIs, such as chatbots and ECAs, and 

to design Graphical User Interfaces (GUIs) that use the 
metaphor of the conversation. 

The third point is that the ability to converse strictly 
depends on the conversational agent. Therefore, 
products with built-in assistants will have the most 
advanced capabilities. Products that are only "assistant- 
enabled", without any impact on the product, can hardly 
be considered as conversational. 

The fourth and last point is tangibility. There are 
different elements that show potential for tangibility: 
the design of the “invocation” of the agent, the design 
of the system's outputs, and the design of systems to 
guide users in remembering and visualizing the available 


actions and commands. 


Further research could explore the firsthand experience 
of designers that are already involved in related 
projects, with the goal of investigating their role in the 
development of ConvSP. Other activities could include 
practical tests with ConvSP, and design activities aimed 
at exploring how physical attributes could work in 


synergy with the verbal dimension. 
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Al-to-Microbe Architecture: 
Simulation, Intelligence, Consciousness 


Abstract 


This paper probes questions of how big machines— 
buildings—can function as hybrid metabolic/Al 
organisms. Focusing on Al, artificial life (ALife), and 
microbial intelligence | look through the lens of Ludwig 
Wittgenstein’s Tractatus and Alan Turing’s algorithmic 
plant simulations to source modernist theory for 
biointelligent architectures. I’m using scant records and 
testimony interpreted through each thinker’s writings, 
architecture, and/or simulations. This text is then a 
device for considering ways-of-being within, and ways- 
of-thinking about, theory/practice for the fusion of 
biological-to-biosynthetic intelligences (microbes, plants, 
animals, Al, machines.) Resulting theory thereafter 
supports the development of bioremedial environmental 
cleanup addressing climate change. My proposition then 
deploys biomimetic and laboratory data to nurture 
metabolically driven intelligences partnered with Al 

in the production of architectures. That ontological 
pathway stems from machine learning, bio-surveillance, 
and digital simulation at object, agent, and urban scales. 
Accomplishments in neural net Al and synthetic biology 
stirred me to question earlier breakthroughs in relation 
to current experimental practices. Subsequently, | link 
and hybridize emergent design proposition to Al, ALife, 
and biological intelligences as unities for environmentally 


performative, intelligent buildings. 
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1. Propositions Toward Metabolic 
Architectures 


Research strategies to regulate exploratory sets of 
generative design actions are called upon here for 
reasoning the inclusion of Al, synthetic life, and bio- 
algorithmic generation into the production of metabolic 
architectures. These exploratory tactics underwrite 
hypothesizing biointelligent buildings as parts of nature. 
Therefore, to link theory and observation | evolve 
strategies for the investigation of matter and forces 
starting with symbolic languages to sort types of 


intelligence. 


Specifically, concepts-terms such as “atomic facts,” 
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“form,” “objects,” “substance,” and especially “picture” 
— autonomously inhabiting Wittgenstein’s Tractatus 

— are appropriated for research organization. His 
philosophy suits design analysis reinforced when the 
Vienna house [1] he designed for his sister is decrypted 
to acknowledge Tractatus logic | analyzed in “Calculating 
Turing, Thinking Wittgenstein" [2]. Likewise, for 


this paper, the Wittgenstein House stands as a pre- 


Fig. 1. Brainstorming. Theory for plotting application and methodology for metabolic buildings involving microbe/plant/animal/ 


machine intelligences organized on Wittgenstein's Tractatus and Turing's Morphogen propositions. Drawn for the 2018 Metabolic 


Architectures Studio, UIC. Dennis Dollens. 


computer agent balancing thinking and design practice 
to enact theoretical propositions in architecture. 

In the above framework, Wittgenstein's philosophy 
guides design-research, especially when joined by 
notions of extended cognition [3], extended phenotypes 
[4], and Turing's algorithmic botany [5]. That mix is 

used to prompt theoretical organization to motivate 
Al-to-microbial [6] design conceptualization. Such 
organization enables metabolic and Al linkages through 
Wittgenstein’s logic and Turing’s simulations to illustrate 
relationships for theories of intelligent buildings. In the 
same way, designers may configure objectives for their 
own projects, observations, and strategizing research 


procedures for metabolic architectural [7. 8]. 


The result of Tractatus-prompted reasoning may be 
used to diagram design visualization theory enabling 
architects to consider: (i) the synthesis and mutability 
of life, matter, and forces, (ii) differing typologies of 
intelligence existing in microbes, plants, animals, and 
some machines for (iii) algorithmic simulation. All 
three areas have parallel interactions in Turing’s [9] 
research that complement Tractatu [10] precepts when 
used as foundational logic for Al-managed, metabolic 
architectures. Turing/Wittgenstein reformulations 
thereby suggest pathways over which buildings may be 
designed as living technology [11. 12]. 


Together, Wittgenstein’s propositions and Turing’s 
computational tactics enable design brainstorming [Fig. 


1] to frame programs of investigation. That research 


may then reveal biological attributes of living organisms 
suited to architectural simulations and models. Viewed 
through propositional tactics, the Wittgenstein House 
[1], designed approximately ten years after his book, 

is a built idiom — the logic of the Tractatus resolved 

in tectonic form: book-to-building, mind-to-matter. 
Mind-to-matter scrutiny — mixing modes of autopoiesis 
[13] and extended cognition [3] — further situates the 
house in a relationship with intelligence, language, and 


visualization. 


2. Typologies of Microbe, Plant, & Animal 
Intelligence 


Metabolic architecture — formulated through 
propositions — articulate cognitive states supporting 
mind-to-matter design enactments. It seeks biological 
performance by questioning technological ways to 
extrapolate sensory biointelligence from nature. For 
example, asking: How can designers observe biological 
intelligence to enact architectural bioremediation? And, 
if remedial strategies are sourced in nature: How can 
architects employ theoretical procedures to interpret 
an organisms’ intelligence and appropriate it to monitor 
environmental toxicity? Such questions become 
recursive — referencing history and philosophy to 
evolve thinking and data appropriate to building design. 
The designer may then set objectives receptive to living 
metabolic operations involving the hybridization of Al, 


microbe, and synthetic biology. 
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In this phase, aggregates of Al/microbes may be studied 
as cellular-intelligent agents challenging architects to 
harness carbon dioxide (CO,) sequestration. First 
design explorations are organized through biology 
via autopoiesis (auto = self, poiesis = making) [13] 
interpreted to validate organisms (microbes, plants, 
animals) as intelligent system unities. Theorized by 
autopoiesis, organisms may be composite unities 
coupled with computational technology. Such 
simulations echo Turing's plant observations for 
algorithmic performance [14] and his theories of 


machine intelligence [15]. 


Shadowed by cybernetics, design research is linked to 
biology, biology is linked to code, and code is linked to 
generative architecture. That architecture awaits new 
questions formulated after Turing asked: “Can machines 
think?" [9]. From Turing's starting point, designers may 
fast-forward observations and data to contemplate 
intelligent systems programmed through computational 
biology. In such cognitive-to-computational processes, 
Tractatuslike corollaries emerge as tools [Figs. 1. 4] to 
determine design research methodologies situating 
metabolic buildings as human-extended phenotypes [4. 
16]. 


3. Hybridizing: Nature/Intelligence to 
Machine 


Observation of nature's biochemical processes reveals 
that cellular agents — microbes and plants — can be 
integrated into synthetic materials or ALife provisioned 
for architectural components/facades. The resulting 
metabolic architectures operate as hosts for cellular, 
living organisms communicating between urban 
infrastructures and dynamic ecotones. For such hybrids, 


an architect needs biological data [Fig. 4] visualized and/ 
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or rendered in code to simulate microbial living habitats 
that, until now, have been genetically programmed only 
by nature [17]. Such observations and programming 
presuppose subsystems invoking collective microbe 
behaviors that designers (working with biologists), 
must coax into an alliance with Al in order to ask: Can 
buildings bio-technologically remediate pollution? In 
response, post-Turing questions take for granted that 
intelligent bio-facades could incorporate, for example, 
Al-monitored bacterial colonies (e.g., biofilms) in order 
to convert toxins to energy by feeding on CO, in ways 


pioneered by oil-spill cleanup. 


The thinking behind botanic algorithmic programming 
and the exchange of metabolic data will enormously 
increase when Al and synthetic life are genetically 
cooperating and reproducing in living matter found in 
nature [6. 26]. Existing examples preview beneficial 
bacteria living in animal architectures built of beeswax, 
wasp paper, biofilms, termite mounds, and human-made 
adobe. Biocellular-Al may equally be modeled upon, or 
paired with, microbes and plants to take residence in, 
and perform from buildings and urban infrastructures 
that track pollution while metabolically consuming 


specific toxins. 


4. Turing/Wittgenstein 


By extrapolating from Turing's theories and 
programming, | repurpose biology-to-code 
investigations through which he simulated aspects of 
matter, life, intelligence, and machine processing [5. 7]. 
Procedurally, the lineage stems from his observations 
of living organisms extended to implant functions from 
nature into coding. This meshes with how Wittgenstein 


[10] used propositions to argue the "case" as the 


world and, in this text, when the Tractatus is culled for 


Fig. 2. L-system 
Grown Plant/Microbe 
BioTowers. Left to 


Right: PagodaTower, 


Biolower, and 
icrobeTower, 


Barcelona. Dennis 


Dollens. 


Fig. 3. Left: feroxTowers are theoretical experiments whose function investigates hybrid Al/microbe atmospheric carbon capture 


collaboratively enabling options for bioremediation to produce energy at levels of cellular homeostasis. 2018-ongoing. Right: 


Arizonalower STL & Animation Sequence. L-system plants generated as roots, branches, and seedpods — with the seedpods 


programmed as polysurface rectangles and the roots grown into a branching superstructure. Dennis Dollens. 


design research logic. Extrapolated data, or mind's- 
eye pictures in Wittgenstein's sense, are then available 
to researchers for cross-system investigations in 

which biochemical signals appear as potential codes 
for heuristic Al to learn or decrypt. In my case-world 
experiments, algorithmic generation and morphological 
materialization [Figs. 2. 4] merge to help analyze 
teaching/research pathways over which metabolic 
architectures may source intelligences from nature for 
use in buildings [7. 8. 14]. 


5. Biological Observation as a Design 
Operation 


Design research observations, channeled through 3D 
scanning and SEM imagery [Fig. 4] thus parallel the 
Tractatus' first line: "The world is everything that is the 
case" [10]. Designers evolve individual case-worlds when 
data is retrieved from organisms and applied to design 
— inarguably a realm of human cognitive nature. In this 
situation, scientific procedures supplement material and 
environmental data to support design research. That 
research facilitates inducting ALife functionality and 
behaviors into building materials. Following preliminary 
designs — data and imagery from microscopes and 
scanners [7. 8] — further detail the translation of 
insights [Figs. 1. 4] from nature in order to program Al/ 
microbe material candidates into fablab productions 


and/or agents [Fig. 3]. 


To unpack the above paragraphs requires us to: 

(i) theorize technology, ecology, architecture, and 
computation in terms of bio/synthetic and metabolic 
propositions [Fig 1]. Doing so positions us to 
conceptualize Al enhanced with animate intelligences 
targeted for generative architecture while (ii) 

tasking resultant theory to support programming 
bioarchitectural homeostasis engaged in climate, soil, 
and water restoration. Those two processes integrate 
networks of living (cellular organisms) and Al to (iii) 


envision metabolic intelligent buildings [Fig. 2]. 


The above bio-to-building dialectic [7. 8] enables us 

to characterize intelligent architectures as potentially 
sentient and autonomous. One possibility is then 

to capture airborne CO2 through the actions of Al/ 
metabolic machines performing intelligent analysis of 
toxins executed by microbes in biomechanical systems 
[Fig. 3]. Sentience, routed to architectural functions, 
algorithmic simulations, and cellular performance 
illustrates how metabolic machines could give rise 

to new species of design and architecture [Fig. 3]. 
Emergent propositions thereafter expand territories of 
experimental theory in queries such as: If metabolic/Al 
buildings sense, experience nature, and make decisions 
through collective microbial life, do they experience 


artificial consciousness? [18. 19]. 
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Fig. 4. Datura ferox Data/Image Sets. Top: Scanning Electron Micrographs of Datura ferox seedpod spikes. Middle, left: Datura ferox 


dried seedpod followed by two images of 3D-CT scans. Bottom: Rhino3D screen shots. Imported CT scans for carbon-capture pod 


development of (See: Fig. 3 Left). Dennis Dollens. 


6. Al Anticipates DNA Turing Machines 


From such interrogations — not frequently probed, yet 
lurking behind, for example, Google's AlphaGo, [20] — I 
anticipate ontological nature-to-machine unity. Such 
questions are credible after Google's Al succeeded at 
learning, playing, and winning Atari video games, beating 
world champions at the Game of Go, and triumphing 
over human and machine chess players [21. 22]. Still 
other questions arise because programmers do not fully 
understand the learning processes their codes engender 
in machines. We/they may ask: Are some species of Al 
existentially thinking? [23]. Associatively: Is a subset 

of neural net Al approaching cognitive abilities? — 


abilities humans have traditionally considered exclusive 
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to themselves and, in a lesser register, to a few animal 
species [19. 24]. 


These questions do not suggest a one-to-one microbe 
or plantlike parity with deep learning or reinforcement 
learning Al. Rather, | point out quorum sensing 

[17] potential for genetic biotechnologies [12] to 
prompt computationally originated sensing in living/ 
synthetic cells capable of next-generation inheritance 
and reproduction. (According to Interface: The Royal 
Society — prospects include theoretically successful 
DNA Non-Deterministic Universal Turing Machines 
[25. 26. 27]. Such molecular-scale machines would 


be compatible with quorum sensing and neural net 


Al and could eventually be edited into living cells to 
enable DNA computing [26. 27]). Here then, metabolic 
buildings with active DNA intelligences, could be viewed 
as biosynthetic Al/ALife agents achieving microbe 
cognition long after Turing, but still consistent with 

his algorithmically simulated drawings and theoretical 


writings [5]. 


To be clear, | am saying that Turing/Wittgenstein 
theoretical propositions may now be considered 
members of intelligent nature. They are phenomenal 
agents-of-thought licensed in philosophy, mathematics, 
and cognitive science [3]. Propositions of this 
phenomenal order (metabolic/living/Al intelligences) 
figure as agents of thinking [28] — facilitators from 
human cognition as it evolves new typologies of 
intelligence — in this text’s case-world — as metabolic 
buildings and/or cities. Agents-of-thought, manifested 
through the Tractatus, are accordingly selected as design 
axioms for generating prototypes that consequently 


exist as extended phenotypes [4. 16]. 


7. Can Buildings Think? 


However jarring, constructed species — neural net 

Al and ALife [11. 12. 27] — extend Turing’s question, 
"Can machines think?" [9. 14. 28] His question (and 
Wittgenstein's too [29]), if answered positively, gives 
support to the proposition that buildings, as big 
machines, can think. Design-research goals can then be 
perused for ontological unity connecting ways-of-being 
/ modes of debate / types-of-intelligence / responses- 
to-climate change / and requirements-of-design. As a 
result ontological cohesion aligns research with places, 
tools, and nature as aspects of design contextualizing 
the extension of our cognition [3] in case-worlds. 

That process creates a framework for contemplating 
hybridized machines, Al, and (some) microbes as 
environmental sentinels — new species of metabolic 


intelligence and artificial life [12]. 


Metabolic architectures are thus first ideas, then 
propositions (or codes) developed in design from 
deductive exchange between phenomena, material, 
computation, nature and the architect. In this lineage, 
propositional analysis is realized descending, not only 
from the Tractatus, [10] but also from Latin res (thing) as 


in res extensa; as well as idea found in idein (to see) from 


Greek. The quandary — things seen cognitively — or 
cognitive things (res cogitans) — underpins design (ideas, 
propositions, prototypes) embodied in matter/tools 

for thinking about metabolic architectures. Applied 

to Turing, we need only look at his plant-to-algorithm 
drawings and printouts [5] to comprehend that he 
seized on nature's intelligence and physical growth for 


mathematic language and resulting digital simulations. 


For design theory, this text's case-world is in service 

to algorithmic simulation begun when Turing translated 
plant attributes to programming [15. 28]. Simulation 
then enters our framework, not only through Turing 
but also through Wittgenstein's term to "picture" (res 

* idein). Similarly, propositions can be generative orders 
of simulation — Latin simulationem, simulare to underpin 


thinking and designing as thought prior to coding. 


Turing simulated parts of nature in computation 
appropriate to philosophical design/machine/intelligence 
debate. From that scenario, we confront results of the 
verb simulate and the noun simulation to communicate 
ranges of life/cognition. Simulations, for such usage, 
are thus human-extended phenotypes [4. 16]. They are 
concept/objects of thought as cognitive or computed 
numbers realized (built) as thinking machines/buildings 
constructed in the world [3]. In autopoietic [13] terms, 
they are participants in cognitive-to-physical domains 
(the case-world) that here includes Wittgenstein's 
theory of picturing [2. 10] incorporating language 

and design/construction realized in symbolic logic 


(philosophy) and the built Wittgenstein House [1. 2]. 


To give bearing to this paper, | see Turing as the 
agent from whom we learned how to simulate nature 
with algorithms (e.g., his reaction/diffusion theory 

[5. 15]) as computational extensions of our thinking 
[3]. Those lessons later brought fourth code-to- 
simulation languages (e.g., L-systems) for today's output 
whereby seeing (res) and imagining (ideate), translated 
mathematically, drive machines/Al to simulate nature. 
With such ancestry in mind, | evolve models [Fig. 2] 
using methods Turing pioneered. After extending his 
botanic observations for computational biology to 
CAD/CAM, | use laboratory and fabrication machines 
to visualize and build-out data resulting in various 
scientific, technological, and design pathways for 


metabolic architectural practice [14]. 
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Consequently, by subscribing to precepts of Dawkins's 
[4] and Turner's [16] extended phenotypes, | justify 
intermingling observations of organisms, matter, 

and forces. Thereafter, metabolic architectures are 
propositional descendants of Universal Turing Machines 
[25. 26] and/or botanic algorithms [5] theorized in The 
Chemical Basis of Morphogenesis [14] and Morphogen 
Theory of Phyllotaxis [27]. 


Natural functions such as phyllotaxis in fir cones or 
daisies, borrowed by Turing for his programming 

and drawings are now culturally and technologically 
absorbed. However, they may be further recast — 
computationally resimulated and biofabricated for 
metabolic systems (living technology) not possible in 
his time [Fig. 2]. In that context, a paradox appears 
whereby we simulate nature in an act of intelligent/ 
phenomenal nature itself. Subsequently, at junctures of 
observation other questions arise: If neural net Al can 
simulate or partner with life, can it contemplate itself 
as conscious? [19]. Or: Can bioAl convince multicellular 
systems (microbes/plant cells) that its onboard 
intelligence is a part of their living milieu? [12. 17. 18. 
19]. 


8. Conclusion 


Different iterations of neural net Al along with 
biological and synthetic life intelligences — are being 
developed and or decoded in laboratories around the 
world. Plant intelligence and communication predicated 
on signaling and biochemical reaction/diffusion are 
equally subjects of research [17. 31. 32]. We should 
therefore interrogate composite bio/Al systems 

[33] and dialectics [34], not through vague notions 

of sustainability, but through climate stabilization 
collectively underwritten by theories such as 
autopoiesis [13], extended cognition [3], and extended 
phenotypes [4. 16]. Thereby, designers investigate 
nature to pair with learning Al and/or living intelligence 
that address questions of how big machines — 
metabolic architectures — can function like organisms 
eradicating pollution while monitoring violations against 
nature [35. 36]. 
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Envisioning and Questioning 
Near Future Urban Robotics 


Abstract 


Robotic services, which have started to appear in urban 
environments, are going to transform our society. 
Designers of these robots are not only required to 
meet technical and legal challenges, but also address 
the potential social, political, and ethical consequences 
of their design choices. In this paper, we present a 
workshop format with its related tools intended 

for enabling speculation about such possible futures 

and fostering reflection on potential socio-ethical 
implications that might support/oppose these futures. 
We report the results and discussion of one particular 
workshop case, in which the implementation of two 
particular robotic services for a city was envisioned 
and questioned, i.e., surveillance and delivery of goods. 
By discussing the results, we illustrate how such a 
workshop format might be beneficial for setting the 
agenda for a more conscious design of urban robots and 
orienting future research towards meaningful themes 
related to the emerging coexistence scenarios between 


citizens and robots. 


Keywords 


Future Scenario, Urban Robotics, Robot Capabilities, 
Robot Implications, Robot Conceptualizations, Design 
Workshop 
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1. Introduction 


Provided by the advancements in sensor technologies, 
artificial intelligence, and smart materials, designers 
now have the opportunity to take on the exciting 
challenge of working with intelligence as a design 
material that can be used in form giving practices [1]. 
The idea of automata, which has been an object of 
speculation and pretense since the ancient times [2], 
can now be translated into tangible entities that are 
autonomous, intelligent, and might behave out of our 
direct control. The enthusiasm that may characterize 
this emerging design space, however, often tarnishes 
the contingent need for understanding how these 
novel autonomous artefacts and related services are 
transforming our society, and whether the future we 
are shaping correspond to our needs and aspirations 
as community [3]. Attributing reasoning abilities and 
autonomy to artificial artefacts, in fact, asks not only to 
meet technical and legal challenges, but also to address 
the possible social, political and ethical consequences 
of such a choice. In particular, automation and artificial 
intelligence need to be addressed and designed 
responsibly in public environments like cities, which 
are becoming more and more crucial as contexts for 


technological innovation [3-4]. 


The complex nature of coexistence scenarios emerging 


from the diffusion of these artefacts, hence, point 


out the need for systematically envisioning how these 
near futures might look like. In this regard, the design 
discipline can play a proactive role by providing methods 
and tools for supporting speculation about possible 
futures, fostering reflections on potential political 
structures that might support these futures, and 
enabling a more conscious shaping of intelligent and 


autonomous artefacts [5]. 


With the aim of embracing this call to action, we 
present a workshop format developed within the 
context of the "Things as Citizens" research project 
from Delft University of Technology, as an approach 

to investigate near future scenarios of coexistence 
between people and autonomous artefacts in the urban 
environment. In this paper, we refer to these artefacts 
as robots, but it may be argued that the same reasoning 
and investigation may apply also for other autonomous 
or intelligent artefacts that may not be conventionally 
considered as robots, e.g. automated traffic lights. 
Given this intent of encouraging the envisioning of 
possible near futures and reflecting on recurring 

ethical dilemmas, a workshop format for "envisioning & 
questioning" was crafted and used in various academic 


and non-academic events. 


In this paper, we report the results and discussion of a 
particular workshop in which we tackled two robotic 
services for a city, i.e., surveillance and delivery of 
goods, with non-experts. By discussing these results, 
we illustrate how such a workshop format might be 
beneficial for providing a barometer for the "state of 
the people" with respect to the practical and ethical 
considerations of living with robotic services, setting the 
agenda for future projects and orienting future research 
towards meaningful themes related to the emerging 
coexistence scenarios between people and autonomous 


artefacts in the city. 


2. Envisioning and Questioning workshop 
format 


The Envisioning and questioning workshop and its related 
toolkit resulted from a combination of methods and 
knowledge already familiar in fields like participatory 


city making, speculative design and responsible Al. 


The field of participatory city making includes many 
toolkits developed for enabling collaboration among 
different actors, and therefore, was very rich to borrow 
from. We designed a set of materials (in particular, 

the Key Interactions Board, Critical Review Board, 

and Clustering Board) by referring to the consolidated 
design ideation toolkits meant for envisioning scenarios 
(e.g. [6-7-8]) and workshop formats for facilitating 
dialog [9-10] and eliciting values [11-12] among different 


stakeholders. 


We involved speculative design in the customization 

of the materials and the workshop format to enable 
reflection. As mentioned earlier, our work is dedicated 
to the investigation of possible near future cities in 
which people will cohabit with robots. In particular, 

we refer to the work of Auger [13], who suggests that 

a crucial aspect for crafting speculations and dealing 
with the domain of the possible is to create perceptual 
bridges. These consist in a carefully crafted combinations 
of audience's perception of the world and the fictional 
elements, which can be achieved by designing artefacts 
that are familiar and provocative at the same time. 
Following the author's suggestion, we crafted our 
workshop format by analyzing the context of smart 
cities and its main developments and trends with the 
intent of building a sense of familiarity and plausibility, 
and at the same time, we introduced some provocative 
characteristics in the form of robotic services. In fact, 
we presented the novel automated services as initiatives 
of the Rotterdam municipality, which attributes rights 


and responsibilities to robots. 


Finally, the provocative component used for fostering 
the speculation was defined by current debate and 
literature about the responsibilities related to the 
spread of Al and robotics. We examined the current 
debate about the attribution of rights, legal personhood, 
and citizenship to robots [14-15-16-17] and employed 
these as an inspiration when describing the robots in 
the Things Cards, as well as raised these issues during 


the discussion phase at the end of the workshop. 


The resulting materials and the workshop format, which 
we refer as Envisioning and Questioning workshop, was 
then used in different contexts, including: education 


(for supporting a master thesis about an intelligent bike 
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with agency), an academic conference for facilitating 

the envisioning and discussion of near future robotics 
for children (IDC workshop by Charisi et al. [18]); and 
non-academic conferences for discussing the potential 
opportunities and challenges of robotic solutions for 
near future cities (Border Session Lab, The Hague, 2018; 
Drive Festival, Eindhoven, 2018; ThingsCon workshop, 
Rotterdam, 2018). In the following sections we report 
the procedure and results of the last workshop carried 
out during the ThingsCon 2018 Conference. This 
particular workshop was selected because of its specific 
focus on urban robotic services (main interest of the 
Things as Citizens project) and for the completeness of 


the documentation. 


2.1 ThingsCon Workshop 


The workshop was organized as a two hours activity, 
including an introduction presentation on the theme and 
practical activities carried out by the participants. In this 
occasion, the toolkit was customized to envision and 
problematize two main kinds of robotic solutions that 
are likely to widespread in near future cities, namely 
delivery robots and robots for surveillance (Fig. 1). In 
fact, there are already cases in which these two services 
are already being tested (e.g., Starship Technologies, 
Marble, Dispatch, Knightscope). 


The proposed robotic services were presented through 
the Things Cards, containing a description of the robot's 
tasks and functionalities together with an illustration of 
the robot. These were introduced as an initiative of the 
city, which grants rights and responsibilities to these 
artefacts because of their contribution for the public. 

By emphasizing rights and responsibilities of robots, 

we aimed at addressing emerging problems of social 
roles and possible controversial relationships between 


humans and robots. 


Participants. The workshop was attended by 9 people 
(7 female; 8 aged under 30). The participants were: 1 
professional computer scientist, 1 experienced design 
researcher, 1 young design professional, and 6 design 


master students. 


Process. The workshop was organized in six main 

steps: 

* Group formation and introduction to the robotic services. 
Participants were invited to split in two groups, where 
they received a Thing Card describing one of the two 
robotic services, delivery or surveillance. 

* Ideation of key interactions. The two groups were 
invited to think about what might be the key 
interactions among the robots and people in the city 
when the robot is seen as a member of the community 


with rights and responsibilities. Participants were 


Key Interactions Board / Goods Delivery 


Thing Card 


/ Robots for goods dt 


Thing Card 
/ Robots for surveillance 


D 


Fig. 1. Workshop materials: (A) Things 


Cards, sheets with a description and 


illustration of the proposed robotic service; 


Opportunities 


Critical Review Board / Surveillance Industry 


‘TACIT Things As Citizens ideation Toctkit 
PACT, 


(B) Key Interactions Board, A4 forms with 


white boxes for illustrating and taking notes 


PROS CONS 
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about the possible key interactions and 
situations related to the proposed services; 
(C) Critical Review Boards, A3 forms with 
two sections for pros and cons where 
participants can put their notes sticking post- 
it; (D) Clustering Boards, A2 forms where 


participants were invited to collect and group 


the post-it about pros and cons and take 


notes of the emerging topics. 


invited to describe their ideas through sketches and 


notes on the Key Interaction Boards. 


Presentation and discussion. Participants were invited to 
present their ideas about key interactions to the rest 


of participants who were invited to discuss them. 


Critical review of the services and interactions. All 
participants were invited to review both services and 
the related ideas of interaction. In this reflection phase 
they were invited to take notes on Post-its about 
the opportunities and challenges they could identify 
in each service, taking the perspective of citizens, 
industry or government. The Post-its were organized 
on the Critical Review Boards. 

* Clustering. Participants were invited to take all the 
opportunities and challenges and cluster them 
according to the theme they related to. Participants 
were asked to abstract the specific notes into more 
generalizable topics. 

* Discussion and recap. The reflections emerged during 

the clustering were summarized by two participants, 

one for the opportunities and one for the challenges, 


and the organizers recapped the activity. 


3. Results 


The materials produced through the ideation of 

key interactions (Fig. 2), the critical review and the 
clustering (Fig. 3) were reviewed and discussed both 
in loco with the participants and a posteriori by the 
authors with the intent of extracting insights. The 
two different discussion activities also correspond 


with the two potential functions of this workshop's 


amo eum oF 
mene 


® 
Key Interactions Board / Goods Delivery 


TACIT - Things As Citizens Ideation Toolkit 
PACT, TU Delft 


Discuss and describe through sketches and notes what might be 
the rights and responsibilities of the robot and how these translate in kay interactions. 


results. On the one hand, the participants had the 
chance of identifying and discussing a series of emerging 
opportunities and challenges related to the specific 
cases presented in the workshop. This indicates that the 
workshop format would be useful for setting an agenda 
for collaborative work to be carried out between the 
citizens, government, and technology developers (e.g. 
focused on the actual development of a delivery or 
surveillance robotic service). On the other hand, the 
critical review of the results performed by researchers 
a posteriori was a useful way of generating knowledge 
for research in the area of responsible urban innovation 
and design. By interpreting both the explicit results that 
were reported in the clustering boards and the implicit 
results encoded in the situations represented in the 
ideation of key interactions, it is possible to identify 


meaningful and sensitive topics to orient future research. 


In the following subsections we report a summary 

of the topics emerged from the participants’ work 
and reflection, organized and discussed in three 
macro-themes identified by the authors through their 


subsequent analysis. 


3.1 Data Related (In)abilities 


Participants discussed the robots' distinct ability to go 
to places that humans may not go and see things that 
humans cannot see, using a large array of sophisticated 
sensors and processors, which significantly increases 
their capacity to collect and process personal data. 


They emphasized how, through these data, companies, 


Key Interactions Board / Surveillance © 


Ress me in cate of 
mergen ey YO 
ES 


Charging qrores, heat, 


Discuss and describe through sketches and notes what might be 
the rights and responsibilities of the robot and how these translate in key interactions. 


TACIT - Things As Citizens ideation Toolkit 
PACT, TU Delft 


Fig. 2. Results of the ideation of key interactions. On the left: interactions with goods delivery robot. On the right: interactions 


with surveillance robots. In both cases, participants illustrations show how the robot's role may go beyond its main function and 


provide additional benefits for people. 
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government, and citizens gain new capabilities. 

For example, surveillance robots may increase the 
autonomy and independence of citizens. As envisioned 
during the workshop, children or elderly could be 
accompanied to their homes while being guarded from 
security threats by the robot identifying anomalies in its 
environment. This may provide them greater freedom 
to move around, knowing that they have a personal 


guard that will prevent problematic situations. 


Despite this opportunity, however, participants largely 
stressed how such abilities also come with their own 
challenges, mostly in relation to privacy. Even if the 
surveillance robots are not meant for law enforcement, 
they can easily turn into Big Brother kinds of control, 
not only steering the citizens' activities, but also 
nurturing the feeling of being observed and evaluated. 
The participants envisaged situations such as the robot 
gathering information about citizens from their social 
media and using it identifying potentially “troublesome” 
people, the robot observing people and giving them a 
grade as in the TV series Black Mirror, or the robot 
anticipating crimes by using prediction algorithms and 
machine learning. Or given that these robots might 
well be financed by the governments or particular 
companies, the robot may be programmed to nudge the 
citizens into specific "government approved" behaviors. 
In other words, the ability that the citizens gain regarding 
to mobility may sometimes be hindered by the "disability" 
they simultaneously inherit regarding their privacy and 
freedom. There could be slippery slope towards 


authoritarian robotics, which might cause the citizens 
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Fig. 3. Participants engaged in the 
clustering phase of the workshop and 


final clustering boards. 


deliberately avoiding the robot. 


Although less controversial, the participants also raised 
concerns regarding the delivery robots' potential to 
use the collected data for offering improved services. 
Functionalities such as face recognition to unlock the 
robot, detection of customer location for dynamic 
delivery, access to purchase history for a personalized 
service, and detection of the customer's age as a 
security check to deliver the parcel or not, were all 
envisioned as opportunities for a better service, yet 
come with the cost of invading privacy. Thus, the 
question here is whether the increased efficiency in 
delivery and the improvement of safety are great enough 
to justify the resulting restriction of the individuals' 


privacy and liberty. 


Another issue that was discussed during the workshop 
was related to robot networks and flocks. While 
citizens may think of surveillance and delivery robots 

as single entities that autonomously operate in the city, 
it is more apt to think of them as networked devices. 
They can share the data they collect with other robots 
or control other smart devices in the city, such as traffic 
lights or self-driving cars. The participants envisioned 
scenarios where a flock of robots coordinate their 
actions to “catch” a criminal. However, if the mechanics 
of this data exchange is not understandable, it was 
discussed that the robot-robot communication would 
alienate and frighten the citizens. Again, the pragmatic 
benefits of having robots must be balanced against the 


people's right to data transparency. 


These discussions point out how data, which is often 
referred to as a new design material and loaded with 
large promises for improved services and human 
experiences, may actually became a disabler. Especially 
if associated with the idea of autonomous robots that 
can roam around public environments, the capability 
of collecting and managing data can indeed represent a 


source of discomfort and negative attitude in people. 


3.2 Beyond the Tool Paradigm 
As expected, the workshop highlighted when designing 


robots for public environments, such as cities, not 

only is it essential to make the robot perform its 

tasks effectively, but also to consider other forms 

of interactions that go beyond the mere function. The 
robot, in fact, is usually expected to comply with social 
norms and human habits. What unexpectedly emerged 
from the workshop, instead, is how such non-task- 
oriented interactions becomes prominent if the robot 
is discussed in terms of citizenry relationships. Both in 
the case of delivery and surveillance robots, participants 
envisioned and critically discussed possible emerging 
roles of the robots which, because of their membership 
to the urban community, are expected to perform some 
kind of action that we may consider socially relevant. 
The participants conceived situations that involved 
positive interactions where people felt responsible 

to take care of the robots and vice versa, as well as 
negative ones which could compromise robot safety. 

In the following subsections, we illustrate two types of 
human-robot interactions that go beyond the primary 


robot function: mutual care and self-defense. 


Mutual care. Participants conceived many situations 
in which fully autonomous robots ended-up needing 

a bystander intervention. These situations included 
robots breaking down, getting stuck, or even being 
damaged and vandalized by people. In these cases, 
participants envisioned an active role for human 
bystander who may intervene if the robot calls for help 
or even offer their help voluntarily, such as in a scenario 
where a citizen spontaneously cleans up a robot from 
spray paint. Interestingly, these "help scenarios" only 
occurred in relation to the delivery robots. This may be 
due to the nature of the task of these robots. Delivery 
robots are a service with an immediate practical benefit 


to the citizens, whereas surveillance robots would most 


probably be imposed on them by the government or 
companies (e.g., shopping malls, airports, etc.) evoking 
an uneasiness that comes from being observed and 


judged. 


In many of the scenarios that the participants crafted, 
the robots also "returned the favor" in some ways or 
provided services that can be seen as forms of care 
towards humans. For example, participants envisioned 
situations where a delivery robot carried a first aid kit 
and contacted police in case of an emergency, or offered 
to throw trash in the dumpster after making a delivery. 
Similarly, other participants thought of surveillance 
robots could help tourists to find a place to stay or help 
homeless people to charge their phones or provide 
heat. In these perspectives, the robot becomes a social 


actor. 


Self-defense. The participants also envisioned 
situations in which robots may be exposed to vandalism 
and violence, generating a reason for the robot to 
defend itself, e.g. by contacting security or carrying 
weapons such as built-in taser shocks. Participants 
were mostly problematizing the situation in these types 
of interaction scenarios, rather than taking a positive 
or negative position. In doing so, they compared the 
robots with the security officers, who are allowed 

to carry weapons for self-defense. This comparison 
raised a discussion that questioned to what extent it 

is appropriate for the police to have weapons during 
public demonstrations for example, and if we could 
expect the same right to apply to robots. But in this 
case, the participants reflected on whose legal and 
moral responsibility it will be, if a robot's self-defense 
action turns out to be harmful for the humans or break 


the law. 


3.3 Practical Advantages vs Socio-Ethical 
Challenges 


The discussions during the workshop stressed the 
robots' practical potential for improving the human 
performance in terms of efficiency, reliability, and 
economy, and therefore, improving the quality of life in 
urban environments. Nevertheless, both in the ideation 
and the clustering phase, we noticed how participants 
were counterbalancing the reflections on practical 


advantages with discussions on socio-ethical challenges. 
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The potential practical benefits of the robotic 
services—such as the opportunity for having more 
personalized delivery services or reducing risk 

of violence, crime rate and police budget—were 
constantly counterbalanced by several concerns. For 
instance, a recurring concern was that the possibility of 
having more efficient services might come again at the 
expense of privacy. A particular discussion was focused 
on if the delivery robots should know the content of 
the packages they carry. Some participants considered 
this to be acceptable in some scenarios, such as when 

a robot transporting alcohol is received by a minor at 
the door. Should the robot deliver the contents to him? 
Or, should it be able to detect the persons' age and 
decide accordingly? Similarly, the surveillance scenario 
raised several ethical concerns, especially in relation 

to the possibility of the robot's misjudgment and false 
accusations. Some of the participants created scenarios 
in which the robot pointed at innocent people due to 
its wrong data analysis algorithms, people misusing 

the robot (e.g. a person wrongfully accuses a neighbor 


because of an ongoing dispute), or after being hacked. 


In addition to these issues, the participants also 
pointed out several advantages for companies, such as 
being exempted from following employee rights and 
overcoming issues typically related human employees, 
such as substitution in case of absence or paid leaves 
like pension, maternity, or holiday. But at the same 
time, they stressed how these are directly related to 
prominent issues of partial or complete replacement 
of tasks currently performed by a human being and 


consequent job loss. 


Furthermore, the physical presence of a robot was also 
considered to create challenges for the city and impact 
the physical urban environment. The complexity and 
unpredictability of public spaces such as streets and 
sidewalks can cause malfunctions and collisions, the 
robots can crowd the sidewalks, or the presence of 
robots can force the city to alter its layout of buildings 


and roads for more efficient performance. 


4. Discussion and Conclusion 


The results of the workshop indicated how the format 
met our dual interest in facilitating the envisioning of 


possible near futures enabled by emerging autonomous 
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systems, while fostering reflections on potential ethical 
dilemmas. Both in the ideation and clustering phases, 
we noticed how the participants' discussions moved 
from very practical solutions to critical questioning of 
the possible consequences and controversial situations 
resulting from having robots providing services in 

the city. In a very short time frame, the workshop 
enabled the participants to elicit themes that we can 
acknowledge as relevant and topical for the current 
discussion in the fields of urban robotics and technology 


ethics. 


For instance, the participants extensively explored the 
themes related to human-robot relationships, such as 
(1) human vandalism towards robots, acknowledged 

in academia as a crucial issue [19] which can create a 
barrier to the robots' diffusion in cities [20], (2) the 
open debate on how far the robots should be allowed 
to protect themselves from theft or vandalism [21], (3) 
the robots' ability to perform social rituals as enablers 
of long-term relationships [22], and (4) possible 
relationships of mutual care resulting from the social 


roles played by robots [23]. 


Similarly, the participants' reflections on the potential 
implications of robots also recall topics largely 
addressed in public and scientific debate, such as the 
potential flexibility and customizability of robotic 
services as breakthrough solutions for particular 
contexts (e.g. access to services from very remote 
locations [24]); or possible negative drawbacks for 
people (e.g. being subject to monitoring without 
consent [25], injuries [26], or job loss [27]) which lead 
to discussions about design protocols to prevent them 
[21-28]. 


This effectiveness versus legitimacy dilemma also 
frequently emerged during the workshop in relation to 
data and privacy. This is another prominent discussion in 
the techno-regulation for robotics literature particularly 
about who is deciding what data is collected, with 
whom it is shared, the purposes for which the data is 
processed, and the necessary security measures that 
need to be in place [21-28-29]. 


Although most of this discussion covers topics that 


would be recognized by researchers in the field of 


urban robotics, what is particularly noteworthy is that 
these discussions were not raised by the workshop 
organizers, but by the participants who were experts 
in neither robotics nor ethics (although they obviously 
shared an interest in the topic of the workshop). 

Thus, the workshop resulted to be a particularly valid 
tool for enabling a meaningful conversation among 
non-experts on complex topics. We will continue 
conducting the same workshop among populations with 
diverse backgrounds such as social scientists, ethicists, 
municipal employees and robotics engineers in order to 
capture an even larger sample of issues that might help 
identify new directions in the future of urban robotics 


research. 


Furthermore, in our opinion, the workshop can add two 
additional contributions to the existing debate about 
near future urban robotics. On the one hand, different 
from other kind of investigations that often remain on 
a speculative level, this activity enabled participants 

to ground their reflections on very practical examples 
and to situate ethical concerns into daily life practices, 
rather than extreme situations. On the other hand, 
through the discussion of controversial situations 
emerged from the envisioning activity, participants 
moved from discussions of practical implications to 
reflections on values and societal implications that go 
beyond the specific case of robots. As in the case of 
self-defense which led participants to talk about the 
appropriateness for police to defend themselves when 
engaged in public demonstrations, the problem of 
privacy related to the purchase of particular products 
such as drugs, and the controversial case of delivering 
alcohol to a minor and the question of whether it 
should be a responsibility of the personnel who is 
performing the delivery to discern if it is appropriate to 


deliver a product or not. 


These two aspects, i.e. reflections grounded in 

concrete examples and discussions on the socio-ethical 
implications, summarize the dual nature of designing for 
near futures with autonomous systems. In fact, also in 
the three main themes emerging from the reflection on 
the results, we noticed how considerations of practical 
opportunities and limitations should constantly be 
counterbalanced by reflections on the possible socio- 
ethical impact that the envisioned robotic services might 


have. 


This dual nature, we consider, should be reflected in 
the designer's approach, through a co-presence of 

a pragmatic and an idealist mindset. Designers are 
more and more asked to simultaneously understand 
and deal with new technological capabilities, reframe 
the conceptualizations of artefacts and technology, 
and envision the potential impact of their actions 

on a both specific and societal level. By embracing 
these dimensions, designers can engage in a constant 
dialogue with technical disciplines in order to develop 
a deep understanding of the potentials and limitations 
of technology, and with humanities, especially with 
philosophers and social scientists, in order to abstract 
from situated interactions towards socio-ethical 


reflections and questions of values. 
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Robot Citizenship: 
A Design Perspective 


Abstract 


This paper suggests robot citizenship as a design 
perspective for attending to the sociality of human- 
robot interactions (HRI) in the near future. First, we 
review current positions regarding robot citizenship, 
which we summarize as: human analogy, nonhuman 
analogy and socio-relationality. Based on this review, 

we then suggest an understanding of citizenship that 
stresses the socio-relational implications of the concept, 
and in particular its potential for rethinking the way 

we approach the design of robots in practice. We 
suggest that designing for robot citizenship (in the terms 
suggested by this paper) has the potential of fostering a 
shift from a logic of functionality to one of relationality. 
To illuminate the direction of this shift in design 
practice, we include and discuss three robot concepts 
designed to address and rethink present HRI challenges 


in the urban environment from a relational perspective. 


Keywords 


Citizenship; Design Practice; Urban Robotics; Human- 


Robot Interaction; More-Than-Human City 


1. Introduction 


Current technological advancements in the fields of 


Artificial Intelligence (Al) and robotics have stirred a 


lively debate about the nature of human-nonhuman 
relationships and how these may be regulated. In this 
debate, designers often find themselves caught in 
between sensationalist attributions of citizenship to 
humanoid robots such as Sophia [1], and more pragmatic 
initiatives that contemplate the attribution of legal 
personhood to robots, such as the European Parliament 
Resolution of Civil Law Rules of Robotics [2]. Attempts to 
regulate human-robot relationships through the typically 
human construct of citizenship as a congruent set of 
rights and responsibilities can also be observed in other, 
more mundane cases. The widespread and unregulated 
presence of delivery robots in San Francisco, for 
example, has raised a series of social concerns to 

which the city has responded with a strict regulation 
that limits the number of delivery robots moving freely 
around the city [3]. In contrast, the state of Arizona 

has responded to similar concerns by giving the delivery 
robots the same rights as pedestrians [4] as a way to 


make them comply with the same rules. 


Although robots have yet to populate the urban 
environment en masse, unresolved challenges concerning 
their social desirability and responsibility [5-8] call 

for a deep reflection on what attributing citizenship 

to robots may lead to. This, in turn, compels us to 
rethink future urban environments as more-than-human 
entanglements of human and nonhuman entities and 
needs [9]. 
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While a socio-relational perspective of human and 
nonhuman coexistence has gained attention in the 
field of human-computer interaction (HCI), mainly in 
research concerned with animals and plants [10-13], it 
has yet to make an impact when it comes to the design 
of human-robot interactions. This paper responds 

to this gap with a speculative investigation into the 
idea of robot citizenship. The paper first reviews the 
current debate on the topic of robot citizenship as it 
is underlined by three different rhetorical strategies: 
using human analogies, using nonhuman analogies, 

and pointing to socio-relationality. The paper then 
explores the implications of robot citizenship as an 
instantiation of the socio-relational perspective, and 
in particular its potential for rethinking the way we 
approach the design of robots in the urban context. 
In doing so, the paper attempts to move away from 
addressing citizenship normatively as a codified set of 
rights and responsibilities by proposing an alternative, 
complementary design perspective [14] meant to 
challenge and inspire practitioners to shift from a logic 


of functionality to one of relationality. 


2. Robot Citizenship 


Investigations of the social implications of robots 
through legal categories such as rights, personhood, 
and citizenship, have become increasingly frequent 
within academe [15-21]. In the following sections, we 
draw upon, and extend, existing work on the topic to 
describe three main rhetorical strategies through which 
the concept of robot citizenship can be approached: 
human analogy, nonhuman analogy, and socio- 


relationality. 


2.1 Human Analogy 


The first perspective is grounded in the idea that in 

the future, robots, especially when powered by Al, will 
become so sophisticated that they will be practically 
indistinguishable from humans in terms of cognitive 
abilities, sentience, and self-awareness. In such Blade 
Runner-like scenarios robots may be eligible for rights 
and even citizenship. An extensive argument from this 
perspective is provided by Marx and Tiefensee [18]. 
Although they remain skeptical about robots becoming 
fully sentient, they envision functionally equivalent states 


that would enable robots to perceive and preserve 
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their "wellbeing", making them worthy of protection as 
"moral patients", as Gunkel [21] puts it. Similarly, due to 
their growing complexity and sophistication, robots may 
also be able to detect moral demands from other agents 
and, accordingly, behave responsibly. Because of such 
ability to hold both rights and responsibilities, Mark 

and Tiefensee [18] argue that robots may also become 


citizens. 


Argumentation grounded in possible future abilities of 
robots, however, is often contested [21-23] because 
it relies on overvaluations of the actual capabilities of 
even the most advanced robots. In this sense, such 
argumentation reads more like science fiction than 


plausible foresight [23]. 


2.2 Nonhuman Analogy 


An alternative perspective on citizenship that better 
accounts for current robots' abilities is based on 
attributing citizenship to "useful" nonhumans. Kymlicka 
and Donaldson [24], for instance, discuss animal 
citizenship by focusing on the concept of domestication 
as a qualifying relationship. Domesticated animals, they 
note, can be seen as citizens because of their ability to 
regulate their behavior according to norms of civility 
(thus respecting the rights of other members), and 
because of their ability to perform their duties (thus 
providing a meaningful service for the community). 
Although criticized when it comes to animals [18], 

this argumentation can be applied to robots who 

fulfil the same criteria. There are precedents: rights 
were recently granted to nonhumans by virtue of 

their membership, contribution and relationship 

with the human community. The Whanganui river 

in New Zealand, for instance, was granted the same 
legal rights as humans after a local Maori tribe fought 
for its recognition as an ancestor and a contributing 
member to the welfare and wellbeing of the tribe [25]. 
As Forlano [25] explains, "by granting the river legal 
rights, crimes against the river can now be treated as 
crimes against the tribe". Similar initiatives include the 
attribution of rights to the Ganges and Yamuna rivers in 
India [26] and Lake Erie in the United States [27]. 


However, attributing human legal rights to nonhumans 
may lead to open conflicts between the interests of the 


two, as in is the case of Lake Erie. The attribution of 


rights to the lake, in fact, generated a protest from local 
farmers who claimed that their rights (endangered by 

the impossibility of fertilizing their crops because of the 
protection of the lake) should be anteposed to the ones 
of the lake [27]. Despite the merit of accounting for the 
expanded nature of communities and of stressing the 

importance of contribution to a shared good, then, this 


perspective remains controversial. 


2.3 Socio-Relationality 


While both previous perspectives provide convincing, 
even if controversial answers to whether and how 

we could consider robots as citizens, we argue 

that addressing robots through a logic of rights and 
responsibilities only provides partial answers. As 
Coeckelbergh [22] argues in his discussion on robots 
and morality, by focusing on a robot's individual 
features, the rights approach does not account for how 
relations among entities and the social context itself 
contribute to changes in moral considerations. Instead, 
we could approach moral considerations from a socio- 
relational perspective [21-22] in which morality should 
not be seen as inherent to any single entity but rather 
as an extrinsic quality. What this means is that a robot 
should not be addressed as a moral agent or "patient" 
[28] per se, but as an object of moral consideration by 


virtue of its relations within a social context. 


Accordingly, the socio-relational perspective goes 
beyond individual abilities, and accounts for the relations 
between the individual and the whole. In other words, 
attributing citizenship to robots should not be based on 
the question of whether robots are "like us", or "help 
us", but are "part of us" — a point also made by Marx 
and Tiefensee [18] in their account for citizenship based 
on robot sophistication, and by Kymlicka and Donaldson 
[24] in their discussion of citizenship for domesticated 
animals. Going beyond rights and responsibilities, a 
citizen, to be qualified as such, should be engaged with 
other entities in interdependent relations aimed at a 


collective welfare. 


By firmly shifting the emphasis from a logic of rights 
and responsibilities to one of relations, then, this 
perspective reveals the need for a richer vocabulary 
(that would, for instance, help differentiate hard from 


soft rights [22]), or a completely new one that would 


enable us to account for the different forms of moral 
considerations that arise from new human-robot shared 


performances. 


3. A Design Perspective on Robot 
Citizenship 


The preceding discussion hints at how the concept of 
robot citizenship may help the HRI field to extend its 
interest from technical concerns to social ones, shifting 
its focus from pragmatic and technical challenges to 
topics like relationality and ethics. Given that current 
debates on robot citizenship tend to focus on normative 
questions and seek resolution in policy and regulation, 
a process that tends to react to technological 
developments instead of anticipating them, we suggest 
there is value in considering robots as citizens as a 
matter of philosophical and designerly speculation. In 
other words, we are not interested in offering legalistic 
solutions for the more-than-human city or a critical 
speculation on the near future, but in opening up a 


provocative design space. 


By introducing the notion of robot citizenship in this 
way (in terms of relationality and not legality), we invite 
designers to look at emerging human-robot interactions 
not as a matter of individual robotic capabilities but 

as a matter of the relations among robotic and non- 
robotic entities. Discussing robot citizenship, therefore, 
invites designers to approach the design of HRI from 
considerations of the community, its values and shared 
goals, instead of from the individual robot's functional 
capabilities. Through this conceptual shift, considering 
robot citizenship may not only contribute to the 
ongoing discussion about meaningful future partnerships 
between humans and computational artefacts [29-33], 
but also contribute to a more holistic view of HRI. 

The question remains, how can this conceptual shift be 


translated into actionable design strategies? 


As described above, approaching design from the 
perspective of robot citizenship asks us to rethink the 
performance of robots interdependently, and thus 
investigate the appropriateness of the relationships 
between robots and other entities instead of robots on 
their own. As summarized in Figure 1, the design space 
that opens up in response to thinking about robots 


as citizens requires that we understand how a robot 
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E: 


Conflicting goals 
or inabilities 


may enter into relation with both other entities and 
the environment. From this initial framing, then, it is 
possible to identify opportunities for meaningful and 
appropriate partnerships by reflecting on the possible 
roles that both human and nonhuman entities may be 
asked to perform together, on the basis of what Kuijer 
and Giaccardi [33] refer to as capabilities that are 


"uniquely human" and "uniquely artificial". 


To this suggestion we add a specific perspective: 
instead of looking only at what individual entities are 
'good at', we suggest considering what they are not 
good at. As Marenko and van Allen [31] argue, most 
current approaches to interaction design tend to be 
task-oriented and efficiency-driven, and therefore 
tend to produce specific narratives about devices as 
consistently behaving entities towards which people 
often build inappropriate expectations. By recognizing 
limits and coming to terms with unpredictability, and 
by suggesting narratives of "dumb-smart" [34] entities, 
designers can free themselves from the idea of designing 
for perfection and redirect their actions towards 
"ecologies of things that are mutually responsive and 
interdependent" [31]. 


Once they identify such inabilities, designers can explore 
how the same performance is successfully instantiated 
by other entities, as a way to envision possible design 
alternatives. Among these, we suggest focusing on 
strategies that communicate interdependency and may 
foster values that can be considered appropriate for 
human-robot interactions. This perspective helps us not 
only to pivot towards relationality and interdependence, 


but also to shift our focus to the extrinsic (rather than 
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intrinsic) qualities of a robot that can enable appropriate 
forms of interaction. To do so, we need to understand 
what Coeckelbergh [22] calls “apparent features”, 
according to which the features of a robot are not 
appropriate or morally significant on their own, but only 
by virtue of their interplay with other entities (much in 
the same way that "affordances" differ from technical 
features). Consequently, by addressing this socially 
constructed idea of appropriateness, designers are 
invited to craft robot features that account for how these 
features would be experienced and judged by humans. 


In what follows, we illustrate the design space that 
opens up by considering socio-relationality as a key 
framing for HRI. We start with a brief discussion of 
urban robot challenges, and then present three robot 
concepts that illustrate how addressing robot citizenship 


can translate into tangible design strategies. 


4. Urban Robots in Question 


By approaching the design of robots from the 
perspective of robot citizenship, we developed three 
concepts for urban robots. These address real world 
challenges faced by designers of urban HRI, that 

were identified through interviews with five robotics 
researchers with expertise in autonomous navigation 
for unmanned ground vehicles (UGV) (a type of robot 
used in urban applications, e.g., delivery of goods). We 
started the interviews with a short introduction about 
the project and its objectives, stressing our interest 

in identifying what are the most pressing challenges 

in urban robotics. We then investigated further the 
emerging challenges through a focused review of related 


HRI literature. Then, for each of these challenges, 


Fig. 2. Overview of 
urban robot challenges 
identified through 

the interviews with 


roboticists. 


we suggested an alternative approach to the problem 

by identifying potential cooperative strategies and 
envisioning apparent features that may facilitate them. 
We exemplify the concepts with illustrations that 

show a robot in a specific situation, communicating an 
implicit message, and provoking a desired response from 


humans. 


4.1 Robot Challenges and Relational 
Alternatives 


Our interviews with the roboticists helped us identify 

a small series of current urban robot challenges (see 
figure 2). These challenges include problems related 

to (1) the robot's need for adapting to non-dedicated 
infrastructures and related social norms, e.g. navigating 
sidewalks and adapting to pedestrians' speed and norms; 
(2) inefficiencies resulting from the need for keeping safe 
distances, as a way to appropriately navigate crowded 
areas and deal with the unpredictable behaviors of 
other entities; (3) issues emerging from the robot's 
difficulty of being understood, which may lead people 

to misjudgment and adverse feelings; and (4) inability 

of being respected, which makes the robot a potential 
victim of human malicious actions, e.g. hacking and 


bullying. 


By reflecting on some of these issues and envisioning 
possible alternative approaches, we developed three 


concepts: the Transparent Robot, which responds to the 


ADAPTING TO NON-DEDICATED 
INFRASTRUCTURES 


© KEEPING SAFE DISTANCES 


issue of being (mis)understood; the Handleable Robot, 
which responds to the difficulty of dealing with non- 
dedicated infrastructures; and the Shapeshifting Robot, 
which addresses the challenge of keeping safe distances 


from others. 


The Transparent Robot. This concept (Fig. 3A) 
illustrates situations in which a malfunctioning robot 
may be perceived as something mysterious and 
potentially dangerous, thus generating adverse feelings 
and attitudes in human bystanders. In fact, “if a robot 
is just standing somewhere looking as a generic 

box, without doing anything, people may think it’s a 
bomb” stressed one of the interviewees. Common 
HRI strategies address this issue by preventing and 
detecting malfunctions with regular interval checks 
(e.g. [35]). However, this challenge may be reframed by 
considering the social environment the robot is part of, 
and relying on the human perception of the situation. 
Recalling existing practices (e.g., calling for assistance 

if an elevator breaks; calling the owner of a lost dog to 
bring it home, etc.), we suggest malfunctioning robots 
may be addressed not as a matter of manufacturer 
responsibility, but rather as a case of a community 


member in need of care. 


From this perspective, designers may shift their focus 
from increasing efficiency to evoking empathy. One 


feature that may help facilitate this shift is the robot’s 
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appearance: changing its appearance from opaque 

to transparent (with or without additional symbolic 
elements), the robot can communicate its situation: 
"Look what happened to me! | need your help!" In this 
way, the challenge of human (mis)interpretation of 
the situation may become an opportunity to generate 


empathy and care. 


The Handleable Robot. This concept (Fig. 3B) relates 
to the challenges faced by a robot when attempting to 
fit its behavior to non-dedicated urban infrastructure. In 
such cases, the robot's autonomy is strongly dependent 
on its ability to combine various skills like detection, 
prediction and planning. Take, for instance, the case of a 
robot trying to cross a street or a busy intersection on 
its own — a particularly complex challenge [36]. Within 
current HRI strategies, this challenge is addressed by 
improving the robot's autonomy and self-sufficiency — 
creating better environmental modeling and detection, 
and designing more sophisticated predictive algorithms. 
However, if we look at social encounters in similar 
situations, we notice that the same problem is often 
dealt with interdependencies generated by affinity and 
shared membership in the community. Recalling these 
existing social phenomena, then, we may consider 

the possibility of a robot's lack of autonomy as an 
opportunity to instantiate interdependent relationships. 
Much like children, elderly, or the disabled, robots may 
cross a street safely by joining a shared performance 


and relying on the abilities (and kindness!) of others. 


When arriving at a crosswalk or needing to cross a busy 
street, the robot may communicate its need for help by 
producing a gesture that mimics the way humans reach 
for help by extending their hand to others. This might 
be achieved by rethinking the shape and purpose of the 
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Fig. 3. Rethinking 
design challenges 
through relational 
strategies yielded three 
concepts (from left to 
right): the Transparent 
Robot (A), the 
Handleable Robot (B), 
and the Shapeshifting 
Robot (C). 


flagpoles that sometimes protrude from the robot's 
back. The pole, in this mode, can be used not only to 
signal the presence of the robot, but also to function as 
a steering device that indicates that the robot is flexible 
enough to be helped. Shifting the position of the pole, 
the robot implicitly says to humans: "You can help me by 


handling me". 


The Shapeshifting Robot. This concept (Fig. 3C) 
responds to the challenge of navigating a crowded 
environment, where the robot's difficulty to predict the 
behavior of a large number of moving agents (especially 
people) represents a very complex problem. Currently, 
designers try to solve the problem by developing 
algorithms based on a "preventive approach" in which 
the environment and other entities are detected, 

their behaviors are predicted, and the movements 

are planned for avoiding collision. Nonetheless, the 
complexity of the challenge and the insufficiency 

of current modelling efforts often lead to errors in 
navigation, harm to humans, or robots stopping in 
their tracks in order to prevent harm. As stated by one 
of our interviewees, "most of navigation algorithms 

are designed to be passive [...] there is too much 

focus on prevention". This highlights how current 
design strategies do not relate to the social nature of 
the challenge they address. When we look at crowd 
behavior, however, we notice that it is often regulated 
by a series of tacit norms that go far beyond the desire 
to avoid bumping into others. From body gestures that 
enable a mutual understanding of intentions, to gentle 
physical contact, humans, as well as animals, adapt to 
each other. What we suggest, then, is to look at the 
robot as a constitutive part of the crowd and, as such, 
an entity that can enable such mutual understanding and 


gentle physical contact. 


Providing a robot with a flexible soft shell, for instance, 
may mimic how humans acknowledge and interact in a 
crowd, sleekly squeezing and slipping through the crowd 
respectfully instead of parting it aggressively. Through 
these nonverbal behaviors, the robot communicates to 
its human surrounding: “You can touch me as | go by, I’m 


safe and friendly”. 


5. Discussion and Conclusions 


The examples discussed above hint at practical 
implications of approaching robots not as tools but 
rather as members of a co-performing community. 
Subsequently, designers may be able to solve 

some of the challenges inherent to complex urban 
environments by designing robots that would be 
perceived, recognized, and tolerated as valuable 
members of the community. In this mode, the three 
concepts we describe above replace self-sufficiency with 
interdependency; autonomy with mutuality; and a tool 
perspective with a civic sensibility. 


By considering the community instead of the individual 
robot’s functional capabilities (or lack of), designers 
can gain a more holistic view of HRI, understanding 

a robot according to its embeddedness in the urban 
environment, its social relations and practices. In 

this perspective, what usually represents a limit and 
challenge for a robot, may become an opportunity 

for instantiating meaningful shared performances 

with humans, in which the abilities of one may 

become a strategy to deal with the limits of another 
[33]. Furthermore, by reframing HRI challenges as 
sociotechnical and not merely technical, the concept 
of citizenship helps to unfold a design space that is 
much less reliant on the legal system’s catching up to 
the everyday presence of robots. Thus, while effort is 
being put to regulate robots from a legal perspective, 
designers may already move forward by thinking of 
urban robots as social actors and therefore anticipating 
regulatory and behavioral responses. The design of HRI, 


it follows, can become anticipatory instead of reactive. 


With that said, despite its potential, this new design 
space is not free from complications. First, our proposal 
assumes that citizen robots, or, more accurately, 

robots that behave as citizens, will elicit certain 


responses from humans. But, what if nobody wants to 


take the extended hand of the Handleable robot and 
help it cross the street? What if nobody cares if the 
Transparent robot needs help? What if the softness of 
the Shapeshifting robot is only perceived aesthetically? 
What if people, despite those apparent features, still 
perceive robots as an obstruction, a burden, or even as 


competition? 


This last question introduces a second complication. We 
suggested here that robots may be considered members 
contributing to the community’s common good, but 
who gets to decide what that means in practice, and 
how? Should HRI designers be responsible for deciding 
what robot uses and roles are desirable? Should they 
hold public referendums on each and every proposal 

for an urban robot? And even then, what if conflicting 


proposals emerge from within the community? 


These critical questions highlight how despite its 
practical implications, our approach does not provide 
solutions, but only opens up a larger space for 
discussion. This very ability to raise questions and 
foster further reflections, however, is what we believe 
represents the very meaningful nature of citizenship as a 
concept that can be used in the design and investigation 
of urban robots. In fact, “in certain circumstances asking 
questions is as important as solving a problem” [37]. 
Design can play a crucial role in this. What we suggest, 
then, is to look at citizenship as a design perspective that 
can be used to challenge existing norms and attitudes, 
provoke discussion and question established practices 
[38] — a way to question the drive for technical 
efficiency that characterizes current robot design. 
Through the concept of citizenship, designers are 
invited to question beliefs about the role of robots in 
society, and to rethink their approach to urban robotics 
from the logic of autonomy and efficiency towards 


relationality and interdependency. 
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Towards Transparency Between 
the Autonomous Vehicle and the 


Pedestrian 


Abstract 


This paper addresses the new problem of transparency 
in relation to pedestrians’ interaction with driverless 
vehicles, arising from their lack of visual cues to replace 
those currently provided by the visible behaviours of 
the driver. It reports two observational investigations 
of the affordances of the street, one looking at the 
street as static environment, the other at pedestrian 
behaviours in relation to driven vehicles. The findings of 
the research were used to identify the decision-making 
process, timings and exhibited behaviours of pedestrians 


and drivers in the street environment. 


Keywords 


Transparency, Interaction Design, Behaviour, Street 


Environment 


1. Introduction 


The field of autonomous vehicles (AV) has recently 
received considerable attention with the rapid 
development in the industry both by traditional 
automakers such as Jaguar Land Rover, Nissan and 
Volkswagen, and leading innovators from other 
fields such as Google, Lyft and Uber. In spite of 
the increasing capabilities of autonomous vehicles, 


such as environmental sensing, object detection 
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and compliance with rules, their ability to react to 
unexpected situations is still questionable [1]. On 

a more general level, Zimmerman[2] explains the 
competent use of a given rule. He mentions that the 
usage of rules is dependent to the state of normality 
as the unpredictable occurrence of situations threats 
the production of desired outcomes. From this point 
of view, the deployment of autonomous vehicles in the 
urban environment is still a concern on safety grounds. 
There is evidence that some types of pedestrians have 
low levels of confidence in interacting with driverless 
cars [3]. 


Companies in the automobile industry were said to have 
invested in safety-related technology around $80bn 
dollars by the end of 2018 [4]. Much of the research is 
concerned with low-level interactions, disregarding the 
complexity of the urban environment. Autonomous 
navigation in a busy urban street environment is 
currently a challenge for driverless car innovators due 
to the unpredictability of the bidirectional interaction 
between humans — particularly pedestrians — and 
autonomous vehicles. The research reported here 
contributes toward this longer-term goal through 
research into pedestrians and driver behaviours in 

the existing street context. This is a prerequisite 

for understanding how to design more transparent 
autonomous vehicles whose behaviour is more easily 


predicted. 
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Semcon 
Smiling Car Concept 


Jaguar Land Rover 
yes Concept 


Nissan IDS 
LED Screen Concept 


Umbrellium 
Responsive Light Up Pedestrian Crossing 


Mercedes Benz F015 
Virtual Zebra Crossing 


to move 


Mercedes Benz F015 
Virtual Zebra Crossing 


Mitsubishi 
Forward Indicator 


AutonoMi 
Pedestrian Detection and Tracking Indicator 


Fig. 1. Designs for autonomous vehicles or streets to improve the pedestrian crossing situation. 


The products in the pictures are designed, respectively, by: Drive Al, Semcom, Umbrellium, 


Lumiled, Mercedes Benz, Jaguar Land Rover, Nissan, Mercedes Benz, Mitsubishi, AutonoMi. 


2. Key Concept: Transparency 


We first introduce transparency in our context, then 
overview relevant interaction design centred on 
autonomous vehicles. Research in intelligent systems 
and human interaction shows a clear demand for 
transparency [5]. Much of the debate is around making 
machine intelligence accountable [6][7][8] with emphasis 
on their being transparent after the fact, though the 
need for intention to be perceptible is also acknowledged 
as important, which may include designing intelligent 
machines so that their general appearance allows their 
genre of action to be predicted [9]. Our interest is 

in transparency immediately prior to and during the 
interaction. Kirsch [10] describes the interaction 
between two parties as transparent when the user, 

in our case the pedestrian, with a certain amount of 
information may understand clearly what actions an 
object affords. Each party gains understanding through 
sharing information clearly and intentionally with the 
other [11]. An important consideration is the optimum 
amount of information-sharing to reach a satisfactory 
level of communication [12]. Overloading individuals 
with information is not desirable in many contexts, but 
especially when rapid and effective decision-making may 
make the difference between life and death or serious 


injury. 
2.1 Designs to Increase Transparency of 


Autonomous Vehicles 


When a pedestrian observes a traditional driven vehicle, 


much of that vehicle's imminent action is predictable 


because the pedestrian sees not only the vehicle but 
also the driver. The pedestrian reads the posture, 
gaze, gestures and expressions of the driver. The 
vehicle-driver system taken as a whole is productively 
transparent. If the vehicle is wholly autonomous, 
however, these key indicators are missing. The vehicle 
has become opaque and its imminent actions are no 
longer predictable. In the image above, we can see 
several recent attempts to overcome this problem. In 
the projects of Drive Al, Lumiled and Nissan, messages 
inform pedestrians textually of the vehicle's intention, 
while the smiling-car concept expresses itself through 
a human-like gesture of smiling to communicate with 
pedestrians. Another anthropomorphic imitation by 
Jaguar Land Rover applies moving eyes to the vehicle. 
The Autonomi concept detects and tracks pedestrians 
and re-communicates this data through its LED lights. 
Mitsubishi, Mercedes Benz and Umbrellium concepts 


instead focus on signals from street signage design. 


2.2 Conceptualizing the Problem 


Theoretical insights relevant to the problem include 
those of phenomenology and ethnomethodology. 
Merleau-Ponty's analogy of the player's navigation and 
exploitation of the football field [13:168] is particularly 
relevant to the dynamic, emergent, often antagonistic 
negotiation by pedestrians, drivers, and others, of 

the streetscape. It echoes Gibson's [14] concept of 
affordances and his emphasis on embodied cognition 
within dynamic contexts. Garfinkel's ethnomethodology 
is useful for its emphasis on the emergent production 


of acceptable and effective behaviours, for example 
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Fig. 3. Panoramic Photo of Piccadilly Circus 


his characterisation of understandings "progressively 
realized and realizable through the further course of the 
conversation" [15:41] and the emergence of a "common 
scheme of interpretation and expression" [15:40]. 

As developed by Zimmerman [2], these ideas of the 
dynamic, co-production of behaviours include ad- 

hoc rule-breaking and rule re-interpretation in the 
service of pragmatic goals. Liberman [16] argued that 
pedestrians and vehicles concert their movements to 
form a local orderliness that better solved the problem 


than enforcing traffic rules. 


3. Methods: Reading the Street, and 
Observing Pedestrian-Driver Interaction 


In our first study, we mapped the affordances of two 
environments in London: Exhibition Road (Fig. 2) and 
Piccadilly Circus (Fig. 3). These streets were selected 
because they both attract mainly tourists even though 
the designs of the streets are different: in particular, the 
first is a "shared space" [17] with deliberately ill-defined 
zones for pedestrians and vehicles, while the second has 


traditional limits. 


We looked at the streets in two ways: first our own 
analysis of the affordances of the environment, then 
observation of how pedestrians seemed to "read" 
these affordances. In the second study, we explored 
how pedestrians understand the intention of drivers 
and what kind of non-driving tasks are performed 

by drivers to communicate their intent. The aim was 
to identify the components of natural interactions 
between pedestrians and drivers in the chosen street 
environment, in order to specify the inputs which, in 
particular, leads pedestrians to make decisions about 


crossing or not crossing the street. 
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3.1 First Method: Reading Affordances in the 
Environment 


Gibson used the term affordance [18] to capture how 
the physical state of an object or environment permits 
and encourages particular sets of interactions. A 

key feature of his thinking was the shift away from a 
nominative approach to perception - one based on 
naming and classifying - to a verbal one, based on action 
and the potential for action. Gibson's emphasis was 

on affordances that already exist, whereas Norman's 
later work shifted the emphasis to the deliberate 
design of visual affordances [19]. By identifying existing 
affordances it is possible to understand how, through 
design, we may be able to invite behaviours and to a 
certain extent predict possible interactions around 

a certain object. Knappett [20] describes the key 
elements of affordances as sociality, relationality and 


transparency. 


To explore our two environments and users' 
perceptions of them, we created concept maps 
divided into two parts: direct perception and indirect 
perception. The first refers is our own observation of 
the properties of the environment [18]. The second 
captures aspects of the process of others' meaning- 
making in the space, by evaluating it within live social 
situations [20]. The framework of the map was inspired 
by Ferrarello at al.’s map where physical artefacts are 
connected by wires to labels for affordances derived 
from objective and subjective assessments [21]. This 
framework captured the differences and similarities in 
these two environments, showing how people use the 
street and interact with it. Our findings are discussed 


below at 4.1. 


3.2 Second Method: Using Behavior Coding to 
Capture Interactions 


We conducted live observations of street users in 
context for brief periods of time on multiple occasions, 
and coded their behaviours, using standard coding 
techniques [22], focusing on analyzing interaction [23]. 
During the observations, we periodically summarized 
the physical and non-verbal behaviours of the individuals 
in the specific categories defined in figure 4. Each code 
is used to mark the occurrence of a specific behaviour 
or set of behaviours. The result is a sequential record of 


the behaviours of one or more individuals. 


Fig. 4. Behaviour 
Coding Data Sheet 


Weather: 
Day & Time: 
Place: 


How long they looked each other (sec.)? 
Who made the first move? 1.Driver 2.Pedestrian 
3.Cyclist 4.Other (mention on the notes) 


Age 1.(0-12) 2.(13-16) 3.(16-24) 4.(25-34) 5.(35-54) What non driving tasks driver did during the 


6.(55-69) 7.(70+) 

Gender (M/F) 

Type of User (if definable) 1.Tourist 2.Resident 
3.Office Worker 4.Student 5.Taxi Driver 6.Uber 
Driver 7.Working in Delivery 8.Uniformed Worker 
(number) of People 

Group Description: 1.School Trip 2.Friends 
3.Family 4.Couple 5.Colleagues 

Commuting Types: 1.Driving 2.Cycling 
3.Pedestrian 4.Motorcyclist 

Who does s/he interact with? 1.Pedestrian 
2.Driver 3.Cyclist Age-Gender of them 

What kind of behaviour has been observed? 
1.Active Behaviour 2.Passive Behaviour 
3.Aggressive Behaviour 

Where the interaction happened? 1.Road 
2.Sidewalk 3.Square 4.Crosswalk 5.Bus Stop 
6.Traffic Light 

Speed Change 1.Slowed Down 2.Faster 3.Same 
Did they see each other? 1.P yes D no 

2. D yes P no 3.Both of them Yes 4. None of them Notes 


music 


The approach provides information about the 
frequencies of specific behaviours engaged in 
interactions by a certain individual. It is reasonably 
objective, though open to nuances of interpretation: 
our resources only allowed a single researcher to 
undertake the coding. It allows us to examine relations 
between behaviours, either within individuals or among 
pairs (pedestrian-pedestrian or pedestrian-driver). The 
observations help us answer questions related to social 
interactional processes in the street environment during 


the negotiations. 


3.3 Group of Participants 


The behaviour coding was applied to a total of 102 
pedestrians (43 female, 59 male). There were 2 between 
age 12-16, 16 between age 16-24, 36 between age 
25-34, 38 between age 35-54, 8 between age 55-70, 

2 over age 70 (all ages estimated). The observations 
were undertaken between May and July 2018, on 
typical weekend days and weekdays. We coded the 
interactions of pedestrians during direct observation in 
the street according to the table in Figure 4 and took 
photos as needed. The data in our mappings are based 
on observations in the field over three days at different 
periods of the day at Piccadilly Circus and Exhibition 
Road. The most significant selection criterion was if 
they were involved in an interaction with another road 


user such as a pedestrian or driver. All the selected 


interection period? 

Crossing Diagonally 

Crossing at the midblock 

Crossing when there is no oncoming traffic 
Crossing when s/he sees other people cross 
Crossing when his or her company encourages 
him or her 

Encouraging his or her company to cross 
Crossing mid block because shop or touristic 
attraction is on the other side 

Crossing between vehicles when they stopped on 
the traffic 

Crossing without paying attention to the traffic 
Crossing while talking on the phone or listening 


Crossing even there are obstacles which reduces 
visibility (parked vehicles, trees, buildings) 
Crossing even though there are oncoming cars 
Driver signals through gaze (1), hand gesture (2) 
honking (3) light (4) 


individuals were trying to cross the road without the aid 


of any form of signal or control point. 


4. Findings 


4.1 Findings from Affordances Mapping 


Location 1, Exhibition Road, showed that people have 
an unusual experience of the space. As commonly 
with shared space, users have difficulty "reading" the 
design decisions represented by the environment. For 
example, not having a curb or difference of pavement 
height clearly leads some pedestrians to think the road 
is for pedestrian only, while drivers fail to identify the 
subtle indicators of lines and if they are going in the 
right lane or not. Such ambiguity has been posited as 

a weakness, and also as a strength of shared space on 
the grounds that it may cause all involved to exercise 
greater attention through their attempts to understand 
the situation [24]. 


When we look into Piccadilly Circus, there are 

some very different physical properties compared 

to Exhibition Road. The design of the junction more 
closely resembles the rest of London in terms of 
heights and materials. The organisation of the junction 
is carefully planned because it is a very crowded and 
busy environment. The majority of the users' age 


group is constituted by teenagers and young tourists, 
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Fig. 5. Affordances Map for Exhibition Road 


whilst Exhibition Road mainly attracts older tourists 
or children. The Piccadilly Circus junction is very busy 
because of the traffic flow at almost any time of the 
day, but especially at peak hours, there is a significant 
accumulation of car users, public transport users, 
cyclists and pedestrians. Illuminated high screens and 
neon lights for advertising create a visual distraction, 
potentially taking attention from the busy street 
environment. In addition, the number of signals to aid 
pedestrians is relatively low considering the complexity 


of the environment. 


4.2 Findings from Behavior Coding: 


Overall, pedestrians' actions noted were either 
observing other pedestrians or making eye-contact with 
a driver. Pedestrians' behaviours were goal-oriented, 
adaptive and far from automatic responses, however 
sometimes during the interactions with the drivers, 
their behaviours were built on elements which are 
automatized. Figure 6 summarises the key behaviour 
coding results for both locations for a total of 102 


pedestrians (51 Exhibition Road, 51 Piccadilly Circus). 


Fig. 7. Results of Behaviour Coding 
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Fig. 6. Affordances Map for Piccadilly Circus 


The Willingness to Interact. 85% of individuals 
were observed looking for signals of the drivers' 
intention to stop or not. They were trying to ensure 
that their commitment to action — to cross the road 

— is appropriate. Clearly, most of the people observed 
wanted to have feedback or respond before they acted. 
Throughout the interaction process, 9276 of the people 
were able to make use of sound information; the other 
7% were either listening to music or talking on their 
phone. 1 person out of 102 clearly had poor vision; he 
managed to cross the road using sound and the help of 


his companion. 


Trust in Collective Behaviour. 42% of pedestrians 
who were observed planned their crossing of the road 
by looking at the behaviour of other pedestrians. They 
seem to put their trust in other pedestrians rather than 
relying on their own individual judgements of drivers or 
of the street system. This behaviour occurred mainly 

in Piccadilly Circus, with 34 individuals. This socially 
constructed engagement with traffic is a relatively 


neglected aspect in the literature. 


Time Course of Interaction 


Exhibition Road 


43 out of 51 crossing 
behaviour occurred 


40 out of 51 pedestrians used 


16 out of 51 pedestrians interacted with another individual for 
maximum 5 sec. and decided to cross or not to cross the road. 


individually. Majority of 
pedestrians crossed where 


speed either as a data coming 
from a vehicle or as a 
communication tool with 


there are no close vehicles to a driver. 


themselves. 


21 out of 51 pedestrians interacied with another individual 
through eye contact for 5 to 10 sec to make a decision. 

14 out of 51 pedestrians interacted with another individual 
through eye contact for more than 10 sec. 


P iccadilly Circus 34 out of 51 crossing 
behaviour occurred 
collectively. Majority of 
pedestrians crossed through 


the moving traffic. 
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33 out of 51 pedestrians used 28 out of 51 pedestrians interacted with another individual for 


speed either as a data coming 
from a vehicle or as a 
communication tool with 

a driver. 


maximum 5 sec. and decided to cross or not to cross the road. 
13 out of 51 pedestrians interacted with another individual for 
5 to 10 sec. and decided to cross or not to cross the road. 

10 out of 51 pedestrians interacted with another individual 
through eye contact for more than 10 sec. 
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Negotiation through Speed Change. A majority of 
pedestrians used the speed of the vehicle as inputs to 
judge when to cross; they also negotiated with drivers 
through adjusting their own speed. The increase or 
decrease in vehicle or pedestrian speeds had a decisive 
effect on the negotiations between them. It was used 
as a means to show an intention to the other party. For 
72% of individuals, this behaviour had an enacting effect 
on the opposite individual. For example, the driver 
reduces speed to allow a pedestrian to safely cross the 
road or the pedestrian takes a step back while trying to 
cross to allow the driver to pass instead. An important 
detail of this interaction is timing and making sure that 


the opposite party can read one's intention. 


Time Course of Interaction. Some individuals 
carried out a series of actions related to the 
consequences of prior actions. For example, 2076 of the 
individuals who were evidently in a hurry performed 
more active behaviours during the interaction period 
while only 18 of them performed an aggressive 
behaviour through sounding the horn at the pedestrian 
several times and making certain hand gestures. On 


the other hand, the results they received were fast 
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speed change) 
Pedestrian Driver 
4. Evaluation of data 5. Action 5 6. Evaluation of Intention 


MEUM the intention) 


2. Uncontrollable Inputs 
(signs, signals, driver's speed, 
other pedestrians) 


3.Controllable Inputs 
‘Pedestrian’s speed and position) 


Fig. 9. The Interaction Process of Crossing Behavior 
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reactions. These interactions were grouped as less than 
5-second interaction in Figure 6. Groups comprising 
families or couples showed more passive behaviour. 
They prioritised safety and tried to cross when there 
were as few vehicles as possible. Their waiting time and 
attention to the vehicles were noticeably higher than 
the rest. Drivers were more cautious towards these 
groups of pedestrians. The duration of interaction 
takes more than 10 seconds compare with individual 


pedestrians. 


5. Discussion: Decoding The Information 
Flow Between the Driver and Pedestrian 


The data gathered through the behaviour coding 
consisted of the reactions of pedestrians and drivers in 
two different places and their negotiation in the existing 
system of a street. The relationship between driver 

and pedestrian behaviours was observed. A conflict 

of interest is clearly indicated where each wishes to 
make progress at the expense of the other. In Figure 

7 we have summarised the elements of a pedestrian's 
perception which affect interactions between pedestrian 


and driver. 


The diagram captures the range of affordances that 
pedestrians use to interact in the street and the 
elements that affect their experience. Even without 
language or digital technology, the affordances of the 
environment inform pedestrians what actions are 
doable and preferable. This information can reframe 
the interaction design between autonomous vehicle and 


pedestrian. 
In figure 8, we created a framework to informs the 


process of the pedestrian's interaction with the driver, 


and how it can be affected by the environment. This 
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helps us to gain understanding about the pedestrian's 


expectation in the interaction. 


We have identified how interactions occur in the 
traditional street environment (Figure 9) and how these 
might occur with autonomous vehicles (Figure 10). 

The diagrams show how conceiving the design task as 
one of constructing affordances, informing possibilities 
for action rather than only explicitly directing the 
pedestrian. This can inspire a new way to design 
transparent interaction between autonomous vehicle 


and pedestrian. 


6. Limitations and Further Research 


This study was part of a larger research project focused 
on designing a transparent framework for interaction 
between autonomous vehicles and pedestrians. 

The framework was evaluated by participants of an 
exhibition using a virtual reality simulation. A future 
step would be to test variations of the framework 
through experimental tasks with a range of timings and 


speeds. 


The study described in this paper focused on just 

two streets. Though different from each other as 
explained, they do not represent the full range of 
street types. It would be also helpful to use technology 
more extensively such as recording the selected 
streets for periods of time and analysing the timings, 
speeds and movements precisely. For instance, 
ethnomethodological video analysis can be considered 
as one of the methods as well. However, even though 
there are gaps that can be filled through more 
observation, we did acquire sufficient information to 
develop a new approach to designing transparency for 
future interactions between autonomous vehicles and 


pedestrians. 


7. Conclusion 


This study is preliminary research towards designing 
transparency for interactions in an urban street 
environment. It contributes to identifying a step-by-step 
approach to the decision-making process for designed 
transparency in interaction. In particular, it highlights 
the importance of feedback, the iterative perceive-act 
cycle, and the need for driver-vehicle system to act 


as a source of actionable affordances for safe action. 
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As a first step, we defined what streets enable right 
now and what are their interactive elements. Then we 
looked into the interactions between actors in the same 
environment. These helped us to understand what is 
the level of transparency in the interactions happening 
in the street environment, what people are looking for 
when they need to communicate with other individuals 


or interact with other objects. 


This qualitative research builds a case for designing 
interactions with autonomous vehicles by considering 
the street environment itself and the existing knowledge 
of pedestrians while taking decisions. The key findings 
are that the influences on pedestrians' decision-making 
process are: collective behaviours, subtle timings of 
sequenced interactions, and the use of speed as a means 


of communicating intention. 


This research points at the opportunity of utilizing 

the interpersonal trust between pedestrians to help 
create a transparent interaction with autonomous 
vehicles. This can be done by triggering this kind of 
collective behaviour through the physical environment 
or the inputs of an autonomous vehicle, as well as 
creating a perception of collectivity in the action of an 
autonomous vehicle. Also, speed change and interaction 
time frame can be used as an input for designing 
interaction with autonomous vehicles. This research is 
presented as an invitation for future work to extend the 


models given. 


Fig. 10. Types of 
Interactions Occurred 
During The Behavior 
Coding 


Fig. 11. The Type of 
Interaction Available in 
Current Autonomous 


Vehicles 
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The Coerced User and 
the Era of Smart City Dissonance 


Abstract 


This paper discusses the societal impact of increased 
connectivity and innovation on the smart city’s 
inhabitants and its effect on the definition of usability. 
We start by discussing the smart city’s connectivity 
revolution and the way it affects the perception of 
usability. In so doing, we eliminate the concept of 
Non-Users and employ instead Coerced Users, who do 
not wish to use the innovation, but are coerced into 
participating—providing it with physical space and 
data—and therefore enjoying the services returned 

in the form of city optimization. We then discuss the 
need for new design approaches addressing these users 
that may be translated into innovation acceptance. We 
present a human-centered design study on the Coerced 
Users of shareable electric scooter services in Tel Aviv. 
It demonstrates the importance of Coerced User design 
and its impact on the inhabitants’ wellbeing. We found 
that the Coerced User’s rejection of innovation is due 
mostly to low-value technology implementation in 

the complex smart city structure, creating a feeling of 
injustice in public goods distribution and an ambiguous 
feeling of “Smart City Dissonance” that affects the 


inhabitants’ relationship with the public sphere. 


Keywords 


Smart City, Coerced User, Innovation Acceptance Life 


Cycle, Micro Mobility, User Experience 
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1. Introduction 


We, as consumers of the public space, are exposed to 
a large amount of technology. This is due to the “smart 
city revolution” which uses connectivity technologies 
and data-optimization software to make our city 

safer and more efficient [1], thereby increasing our 
wellbeing [2-4]. As the smart city is based on data and 
connectivity [2-5], it requires a societal change in order 
to provide the public with its promised value. The city’s 
inhabitants (residents, workers and visitors) must be 
digitally connected—a change that impacts their way 
of life. In return, they enjoy the city’s optimization and 
benefits. With the exponential pace of technological 
advancement [6], this informed transaction will likely 
become a core assumption of participating in a city's 
ecosystem—a new rule that connects personal 


connectivity with being a part of the city. 


We are interested in the impact of such innovation 
entering the public sphere on the relationship 

between technological advancement, public space, and 
inhabitants' behavior and wellbeing. The main problem 
we examine occurs when technology is adopted [7] 
without the public's consent—in many cases imposed 
on inhabitants as a kind of new, unwritten set of smart 
city rules. The public Wi-Fi pole, CCTV cameras, 
drones, and shareable electric scooters are redesigning 
the public space; all of these solutions are designed 


according to the needs of their Active Users—users who 


employ these solutions. But being placed in the public 
space, they have an impact on all the city inhabitants— 
users and non-users alike. As consumers of public 
space, inhabitants cannot avoid these technologies; and, 
like passive smokers, they cannot avoid their impact. 
Gradually, the public loses its option to opt out and 


"not use." 


The connected inhabitants of the smart city are not just 
losing their ability to stop using, but actually provide 
the city's ecosystem with data and physical public space 
exploited for the very services impeding their desired 
consumption of the city space. These inhabitants are 
coerced into participating in the process, becoming 
what we will call Coerced Users of this innovation; they 
are not willful adopters of the technology but are 
compelled to accept its existence as part of the city 
ecosystem and are forced to provide it with resources 
even while suffering from its existence. This could lead 
either to acceptance or to protective behavior [13-14], 
as we describe at the Innovation Acceptance Life Cycle. 
They are still considered "users," however, because 
some of the data collected by negatively impacting 
services might in turn be used—by such services as 
these same Coerced Users employ as Active, Passive 
[8-12], or Incidental [12] users—and therefore indirect 
value is created that may have a positive impact on their 


wellbeing overall. 


This conundrum generates many challenges, but it also 
elicits a better, more inclusive design for the future city. 
Empathizing with Coerced Users’ needs and designing 
products accordingly will become an important part of 
design for the connected world, generating inclusive 
innovation and technology that will reduce, rather than 
increase, inhabitant anxiety and stress, and will elevate 


Coerced Users’ wellbeing. 


In elaborating on the Coerced User, we will review the 
current literature’s definitions of users, expounding 
upon the adoption, acceptance and rejection of 
innovation and technology. We will then proceed to a 
contextual design investigation, using shareable electric 
scooter services as a case study. This human-centered 
work includes field observations; an electronic poll (68 
participants); eight in-depth interviews with experts 

in innovation adoption, public space design and Users 


Experiences; and twelve in-depth interviews with 


Coerced Users. The paper concludes with insights and 


directions for follow-up research. 


1.1 The Smart City and the Always-Connected 
Inhabitants 


The “smart city” is a city ecosystem that uses 
Information and Communication Technologies (ICT), 
Internet of Things (loT) sensors, and optimization 
software to make itself cleaner, safer, and more efficient 
and connected [1-5], thereby providing value-added 
features for city services and generating elevated 
wellbeing for its inhabitants; i.e., its residents, workers 
and visitors. Today, in order to enjoy and benefit from 
many innovative services, inhabitants must be digitally 
connected; and, knowingly or not, share ever more 
information about themselves with the city’s services, 


revealing their locations, habits, and needs. 


As we move toward a realization of the smart city 
vision, it becomes ever more difficult to enjoy the city’s 
benefits without being connected; and, eventually, 

the unconnected may find it impossible to live in the 
smart city at all. This increased connectivity and the 
obligation to share information—the new, unwritten 
rule of the smart city—raise concerns regarding privacy, 
data security, safety, and ethics. Prior work suggests 
that these problems can be addressed via holistic 
co-design approaches for city services, involving all 

city stakeholders [3]. Our research raises a different 
question, however: As a precursor to the design 
process, in this era of always-connected and self- 
optimizing cities, we must first understand the identity 


and characteristics of the “smart city users.” 


1.2 Usability Definitions 


Current designations of users relate to the “activeness” 
dimension in user-technology interaction. Four kinds 
of users have been defined: (1) Active; (2) Passive; (3) 
Incidental; and (4) Non-User [8-12]. Users are divided 
into “active” and “passive” process operators [8]. An 
active process operator’s work differs from that of a 
passive operator by the predominance of monitoring 
[9]. The user’s “activeness” is a result of task allocation 
between herself and the technology [10]. Active Users 
have different kinds of interactions with the technology, 
from operation to maintenance, while Passive Users 


merely monitor the technology and lack control over 
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it. The Incidental User has an interest in the information 
output of the technology but has no control over it [11- 
12]; her communication with the technology is mediated 
by an Active User. Finally, the Non-User simply does not 
use the technology—she either does not want to, does 
not know about it, or uses a competitor instead. For 
example, during an Ultrasound the doctor operating 
the machine is an Active User. The senior doctor 
monitoring the operator is a Passive User. The watching 
parents are Incidental Users. Parents who distrust the 
modern medical system and do not use these tests are 


Non-Users. 


1.3 The Extinction of Non-Users 


Today, when individuals choose to opt out and not use 
a technology, they can easily become Non-Users. This 
situation is changing, however, as city services become 
fully and perpetually connected, disenabling individual 
non-use. In the not-too-distant, fully-realized smart city, 
all inhabitants will be connected and share data with the 
system, enjoying the optimization of services enabled 
by the city's connectivity regarding traffic, bureaucracy, 
and more. They supply data to all services, even the 
ones they do not use; and enjoy the optimization 
derived from that data which is the source of the same 
un-used services. They can no longer not-use; they are 
part of the optimization ecosystem, and even if they 

do not want to use a technology, they are coerced into 
using it as part of the city system. This situation brings 
about the extinction of Non-Users who, consequently, 


become what we have termed Coerced Users. 


1.4 The Coerced User and the Unwitting User 


The Coerced User is a connected inhabitant of the 
smart city and—willingly or not—provides it with data 
while enjoying its optimizations. Even if she decides 
not to use some of the city's innovative services, she 
cannot avoid being part of its ecosystem. First, because 
these services are in the public sphere, and she cannot 
escape their indirect influence - much like a passive 
smoker. Second, because she cannot stop providing 
the city ecosystem with information that might benefit 
these unwanted services, and third, because she cannot 
choose not to benefit from the optimization generated 
by the data these services provide, which might be 
translated into optimization of the services she does 


consume. 
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For example, a Coerced User cannot simply opt out 
and not use shareable electric scooter services—to 
both her disadvantage and advantage. On one hand, the 
scooters still occupy her space, and she needs to dodge 
them in the street. On the other, she also benefits from 
the data they provide the city about the most populated 
routes—data that might influence the city's public 
transportation schedule or infrastructure maintenance 
timeline, and thus have a positive impact on her life. 
Unlike Coerced Users, who are aware of the technology 
they try to avoid, some users may not be aware, but 
still provide data and enjoy the general optimization 
results. These we have termed Unwitting Users—those 
who, for example, unknowingly utilize a public Wi-Fi 
signal, absorbing its radiation while providing it with 


information, such as their location. 


1.5 Designing for Coerced and Unwitting Users 


As the inhabitants become more connected and public 
space more occupied by antennas, security cameras, 
drones, scooters, and more, it becomes ever more 
crucial to understand Coerced Users and to design the 
city experience accordingly—using a human-centered 
perspective. Today, city technologies are designed 
according to a design methodology "correct" for Active 
Users, based on principles of Desirability (attracting 
users), Feasibility (being capable of production by the 
manufacturer), and Viability (being economically viable 
and generating value for both users and suppliers) [16]. 
But since these technologies are deployed in the public 
space, they need to address Coerced Users as well, 


answering their needs and considering their wellbeing. 


1.6 Coerced Innovation Acceptance 


In Coerced User design it is important to understand 
that—unlike for Active Users—service value does not 
translate into Adoption. For Coerced Users, rather, 

it translates into Acceptance. Presently, in order to 
create a successful product, designers need to account 
for the different kinds of Active User—Innovators, 
Early Adopters, Early Majority, Late Majority, and 
Laggards—identified in Rogers’ Innovation Adoption Life 
Cycle [7]. Each of these Active Users must be addressed 


accordingly. 


The same is true for Coerced Users; the designer 


must be empathetic to all the kinds of Coerced User 


when she designs. Coerced Users do not actively use a 
product: they either reject it, if they find it disturbing, 
or they accept it as possibly beneficial for others, 
while consciously choosing not to use it themselves. 
Here we may define the Innovation Acceptance Life 

Cycle that analyzes Coerced Users according to the 
following spectrum of acceptance: (1) The Supporter, 
who supports the innovation but does not use it; (2) 
The Indifferent, who accepts it without supporting it; 
(3) the Soft Rejecter, who rejects the innovation but not 
actively; and (4) the Hard Rejecter, who actively rejects 
the innovation. Each of these must be approached 
differently. The designer should aim for these Coerced 
Users to accept her product, and design the product 
accordingly. Bad design will cause a Coerced User to 
incline toward rejecting the innovation and thus the 
value it might entail for the city ecosystem. For better 
Coerced User design, it is therefore important to 
understand factors that lead to innovation rejection or 


acceptance. 


1.7 Innovation Rejection and the Social 
Amplification of Risk 


The social, psychological, cultural, and economic 
background of a person, as well as media coverage 
and gut feelings, have an impact on her perception of 
technology as a risk, or as something of value [17-21] 
and these lead to acceptance or rejection. Instead 

of occurrent probability and causal intensity, the 
individual analyzes risk according to her psychological 
and cultural filters, using only a fraction of the 
information she receives. To these filters she adds 
possible consequences of the risk and designs a 
mechanism to cope with it. This private and irrational 
action can then become a public approach leading 

to public actions [13]. The Social Amplification of Risk 
theory suggests that an individual's risk perception 
might have an amplified impact. In a ripple effect, she 
influences her community's perceptions of risk [14]. 
The “risk signals" are passed from “mouth to ear,” 
adding to other personal perceptions that reframe the 
risk. This amplification can impact social, political and 
economic structures. In only a short time, innovation 
is framed as risky or otherwise; after this, the window 
of opportunity for acceptance closes. It is important to 
take this into account when designing for the Coerced 
User; for once the public shapes its perception, it is 


difficult to change it. 


2 Contextual Field Research Methodology 


2.1 Contextual Case Study: Shareable Electric 


Scooters in Tel Aviv 


After establishing Coerced User and Innovation Acceptance 
Life Cycle theories, we will now turn to a contextual field 
study in order to validate our theories. This contextual 
field study was designed to understand Coerced Users 
in a transitional, semi-smart city environment, using a 
defined case study representing connected technology 
in the public space. We chose shareable electric 
scooters in Tel Aviv: last-mile transportation vehicles 
spread throughout the city, aiming to provide clean and 
cost-effective urban public transportation. This Micro 
Mobility solution is not a disruptive technology, but a 
unique business model of “use and discard:” the user 
finds a nearby scooter, unlocks it using an app, rides to 
her destination, relocks it, and pays according to the 
time used. The scooters do not require an anchoring 
station; they can be left anywhere. Thus, this service 
represents the main characteristics of connected 
technology in the public sphere: it is (1) an innovative 
intervention in a public domain, which (2) forces the 
user to be connected, since it cannot be used without 
a smartphone and credit card, and (3) is now in the 
interim period that will define its future. These scooters 
represent an interesting case study for Coerced Users, 
as there is a growing, worldwide debate about them. 
They are implemented without any change to the city's 
infrastructure and impact the way pedestrians need to 
act. As the Active Users of the scooters can leave them 
wherever they want, they sometimes create obstacles 
on the sidewalks. These issues generate rejections 
leading to protest and, in some cases, vandalism of the 
equipment. These tools entered Tel Aviv in 2018 and 
their numbers are growing monthly: from ~300 in late 
2018 to ~2,000 in Feb 2019. 


2.2 Contextual Field Study Structure 


Our study consists of interviews, observations, and 
a digital poll. In-depth interviews were held with 
eight professionals from the fields of innovation 
adoption, public space design, and UX. We discussed 
design and innovation in the public space, and the 
phenomenon of Micro Mobility. Observations in 
Tel Aviv included participatory scooter usage and 


shadowing Active Users. We also conducted a Coerced 
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Users observation, learning how they handle / avoid 
scooters. A digital poll was conducted on the way 68 
early adopters, aged 27-50, perceive Micro Mobility. 
This also formed the basis for selecting interviewees. 
In-depth interviews were held with twelve Coerced 
Users about their perception of smart city, public space 


innovation and scooters services. 


3. Findings 


The Coerced Users supported the vision of Micro 
Mobility. This is partly due to a general consideration 
of Israelis as early adopters, positive feedback from 

the media, and the scooter services' promises to be 
easy to use, affordable and "green." Nonetheless, these 
"supporters" of the vision reject its implementation, 
which is imposed on them, invades their public space, 
and negatively impacts their way of life. Our research is 


reflected in the six main themes which follow. 


3.1 Theme 1: “One Fits All” Solution 


To date, the municipality does not require the scooters’ 
providers to modify their product with regard to speed, 
quantity, and location. Accordingly, the providers deploy 
scooters in a “one fits all” approach; some have not 
even translated the safety instructions into Hebrew. 
This situation creates an overload on the city’s crowded 
infrastructure: 91% of the interviewees feel that the 
municipality and providers do not care about the 
inhabitants. The providers “invade” city space without 
talking to the inhabitants or modifying the product 
according to their needs. This creates the impression of 


an unsuitable and dangerous solution. 


3.2 Theme 2: Exclusion of Significant Inhabitant 
Populations 


83% perceive these services as inaccessible to 

some inhabitant populations based on age, weight, 
connectivity, and address (i.e., living where services 
are blocked). This lack of equality in public goods 
distribution inherently creates Coerced Users. Such 
populations cannot use the service but have to pay 
a Public-Space Tax—providing services via sacrificing 


pedestrian public space, getting nothing in return. 
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3.3 Theme 3: Lack of Enforcement 


As these services are owned by private, profit-driven 
companies, 83% feel that providers desire only that as 
many people as possible use their services, no matter 
the risk. There is minimal safety enforcement by the 
providers with regard to inhabitants that misuse their 
platform, and they do not prevent underage inhabitants 
from using it. The safety restrictions presented by 

the providers are perceived as mere prevention of 
formal liability. 7576 feel that the providers' method 

of operation is: "if it is not forbidden, it is allowed." 
Moreover, there is little municipal enforcement, leading 
Coerced Users to suspect a secret deal benefiting 

the municipality and providers at the expense of the 


inhabitants. 


3.4 Theme 4: Feelings of Alienation 


75% of Coerced Users say the scooters arouse feelings 
of alienation. This finding expresses feelings of anxiety 
and a lack of connection between human and space. 

In the words of our interviewees: (1) "It creates a 
feeling of alienation. What is it? Who owns it? It's just 
everywhere." (2) "People just leave it in the street, not 
caring what it does to the relationship between human 
and space." (3) "Wherever we go, when my daughter 
sees a scooter, she turns and says 'Dad, here is your 
scooter. And here, and there.’ This inability to connect 


to a tool depresses me." 


3.5 Theme 5: A Modern “Tragedy of the 
Commons" 


8176 feel that the sharing model encourages Active 
Users to behave in an irresponsible manner that harms 
Coerced Users' welfare; they ride on the pedestrian 
side and park in the middle of the sidewalk. The 
interviewees believe this behavior stems from the 

lack of connection between the Active User and the 
scooter, his desire to maximize the service, and the lack 
of enforcement. These lead to misuse of the service 

by Active Users to the point of harming both the 
scooters and the Coerced Users. The "tragedy of the 
commons" describes a self-interest behavior, contrary 
to the common good, that spoils a public resource. We 
may describe the above as a modern “tragedy of the 


commons." 


3.6 Theme 6: Mental Workload 


7576 describe a feeling of stress when talking about 
Micro Mobility. The service which is meant to engender 
productivity and reduced anxiety for the Active 

User, creates a mental workload for Coerced Users. 
Pedestrians that once wandered the sidewalk with 

ease must now be on constant alert not to be hit by 
scooters. Drivers must also worry about scooters 
popping up, endangering them and getting them into 


trouble with the law. 


4. Discussion 


From the above research and described contextual 
study themes we have extracted three main insights 


regarding Coerced User design. 


4.1 Insight 1 


Coerced Users' problems are due to a lack of 
communication and synchronization among 
members of the triangle of city stakeholders— 
municipalities, service providers, and 
inhabitants—leading to poor implementation. 
(1) The disconnect between municipality and service 
providers, together with lack of regulation, creates 

a burden on the city infrastructure, which is already 
crowded without scooters. The service provider's 

data is not shared with the municipality and thus 

allows neither optimization of public transportation 
where scooters do not operate nor investment 

in infrastructure where usage is greatest. (2) The 
disconnect between municipality and inhabitants makes 
the Coerced User feel that the city is incompetent in 
dealing with the service providers, due to weakness or 
corruption. Coerced Users feel that they are excluded 
from the public space, and that the city—with minimal 
municipal enforcement—cannot help them. It creates 

a sense that "what is not forbidden is permitted" and 
that these "permitted" solutions come at the expense of 
the inhabitants, transforming the city from a place that 
is intended to increase the inhabitants' welfare, to one 
that is exclusionary, and dangerous. (3) The disconnect 
between service provider and inhabitants allows misuse 
by the Active Users and makes it difficult for Coerced 
Users to complain or understand where potential 
solutions might originate. Service providers do not have 


a screening process for users, so there is no connection 


with them when they start using; moreover, there is no 
enforcement of responsible usage, so there is also no 


relation with violators of the law. 


4.2 Insight 2 


Rejection of the innovation is driven primarily 
by a sense of injustice regarding public goods 
distribution; this is reflected in the exclusion of 
some inhabitants who cannot use these services, 
but nevertheless pay a Public-Space and Head- 
Space Tax. The exclusion of inhabitants from these 
services due to age, connectivity, or residency seems 
unfair and raises questions regarding the motives of 

the service provider and the municipality, as well as the 
ownership of public space. These inhabitants cannot 
use the service, but must give up their public space, 
wandering the streets alertly and anxiously while getting 


nothing in return. 


4.3 Insight 3 


The smart city era is accompanied by a Smart 
City Dissonance: an ambiguous feeling with 
regard to belonging, and a double-edged sword 
with regard to efficiency and stress reduction. 
These affect Coerced Users' "head-space." The first, 
Belonging Dissonance, is that feeling of uncertainty with 
regard to the belonging of public space and shared tools 
which—together with a “use and discard" business 
model— encourages irresponsible behavior, leading 

to a modern "tragedy of the commons." The second, 
Efficiency Dissonance, is the negative correlation between 
service effects which are (1) supposed to make Active 
Users more effective (and therefore reduce their 
tension), but (2) by their very existence increase the 


anxiety and stress of Coerced Users. 


5. Conclusion and Future Work 


Coerced Users and the impact of smart city 
connectivity on their lives were at the core of this 
paper. We found that increased connectivity and 
innovative intervention in the public sphere obligates 
us, as designers, to change the ways we view and define 
Usability and Users in the smart city, and the experiences 
that will position Coerced Users on the accepting side 

of the Innovation Acceptance Life Cycle—providing them 
with direct and indirect value and increasing their 


wellbeing. 
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Our results taught us that technology is not the 
barrier for acceptance, but rather its implementation 
in the complex, smart city structure that harms and 
reduces the Coerced Users' wellbeing. Most of the 
frictions experienced by Coerced Users stem from 
disconnections between the stakeholders of the "city 
triangle:” the municipality, service providers, and 

the inhabitants. The primary point of discomfort is 

the feeling of injustice in public goods distribution, 
realized in the exclusion of many populations from 
these services. They cannot use these services but 
nevertheless pay a Public-Space and Head-Space Tax— 
giving up their space and feeling anxious in the public 
sphere without getting anything in return. This paper 
also indicates that the smart city era comes with a 
Smart City Dissonance, an ambiguous situation that 
affects our feelings when going out into the open. First 
is Belonging Dissonance, leading to a modern "tragedy of 
the commons;" second is Efficiency Dissonance, whereby 
a service intended to reduce Active Users' stress has 


the opposite effect for Coerced Users. 


These findings regarding new, smart city users, together 
with the insights of field research, emphasize the 
importance of a holistic and inclusive structure of design 
for the complex smart city ecosystem, and the need for 
a Coerced User design approach and ethic; these will 
increase direct value for Active Users, and the indirect 
value for the Coerced Users, resulting in increased 
wellbeing. This design challenge is an opportunity 

to fulfill the promises of a harmonious smart city 
ecosystem providing benefits, value, and wellbeing to all 
its inhabitants. It is hoped this paper may form the basis 


for future work in this field. 
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Drawing Interfaces: When Interaction 


Becomes Situated and 


Abstract 


The widespread and pervasive presence of devices 
equipped with small-size, small-cost sensors and 
increasing computational capacity affected our 
interactions, making them growingly ubiquitous and 
dynamic; an interaction that shifted from being indirect 
to becoming more physical and direct, as using gestural 
or vocal commands to control smart systems. In this 
context, we developed interfaces that draw themselves 
accordingly to the user need. Drawing User Interfaces 
receive information from the physical world (users and 
environment) via sensors, and react modifying their 


aesthetics and function. 


Keywords 


User Experience, User Interface, Physical Interaction, 


Embedded Technology, Smart Objects 


1. Interfaces, Embodiment and Materiality 


We are in a historical moment of change, where our 
way of interacting with objects is culturally evolving, 
due to the unceasing advances of technology. The 
progressive technological pervasiveness made a 
growing number of devices more intelligent, interactive 


and capable of dialoguing among themselves, with 
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the surrounding context and with us. The result is 
innovative and advanced interactions that produce 
models and patterns of use that reinterpret the 
relationship between humans and technology [1—3]. In 
a context where people and computers are no more 
considered as separate but as a whole [4], our interest 
concerns integrated ecosystems where the physical and 
digital worlds dialogue because of embedded sensors, 
microcontrollers, and actuators that make machines 
"sensitive" to external stimuli [5, 6]. Benefitting of 

a progressive increase in computational capacity 
(ubiquitous computing), such ecosystems can adapt 
automatically to circumstances and decrease users' 
cognitive load [7], namely the information they must 
process to perform a task. Overcoming Graphical 
User Interfaces (GUI), it expanded the range of 
possible interactions: from typing and visualizing (as 
PCs) to manipulation (as Beosound Edge that activates 
a touch-sensitive control interface when the user 
approaches), dialogue (as Amazon Alexa and Google 
Home that can be controlled starting a conversation 
through voice interaction), and gestures (as Xbox Kinect 
that responds to motion sensing inputs, making the 
user's body become the controller). This brought to 

a dual technological advance: the research on Natural 
User Interfaces (NUIs) [8, 9] and on Tangible User 
Interfaces (TUlIs) [10]. In the former case, the logic 


revolves around the absence of a visible interface 


[11], namely user interfaces that are invisible to its 
users, or become so through subsequent interactions. 
The latter case regards embedding the interface into 
the object using manipulation logics typical of the 
analogical dimension [5, 12]. Although we recognize that 
NUIs' immediate and non-mediated interactions are 
interesting and promising, TUIs pop up as even more 
fascinating since they imply analog-like manipulations 
on objects that hide a digital dimension and echo the 
concept of memory of use [13]. However, dealing with 
physical and computational elements that empower 
materials, making them reactive and able to inform 
[14], opens up inevitable reasoning on materiality [15], 
embodiment [16, 17] and affordances [1, 18]. TUls make 
the object on which are mounted both a representation 
of information and a controller [10]. Achieving such 

a total unity between interface and interaction has 
implications as the graspability of the interactions and 
actions supported [19]. Moreover, as stated by Ishii 
himself when criticizing tangible bits in favour of his 
more contemporary concept of radical atoms [14], TUIs 


have limitations in representing change. 


Then, recalling that objects are nowadays more than 
ever shapers of behavior [20], we cannot neglect that 
blending the physical and digital dimensions reconfigures 
the users' behaviour towards the product, impacting on 
the kind of learning such objects demand to be properly 
used [21]. The interfaces we propose advance from this 


very argument. 


1.1 Interfaces that Draw Themselves 


We consider that especially the material perspective 
brings interesting opportunities for the interaction 
design and HCI fields. Looking for potentially meaningful 
aesthetic interactions, our specific interest goes on the 
role that materials, environments and the user's physical 
body as a combination assume in interactions involving 
embodied technologies [16, 22]. From this notion, we 
developed what we define Drawing User Interfaces (DUIs 
henceforth), namely interfaces that draw themselves. 
Such interfaces receive information from the physical 
world (users, the environment, and potentially other 
objects) through sensors, and act accordingly, modifying 
their aesthetics and function. DUIs concern interactions 
with objects where the digital information is translated 


into a manifestation that allows direct manipulation. 


However, it differs from Ishii's radical atoms [14] where 
digital information assumes a physical form. Indeed, 
whereas the physical interaction is maintained, it goes 
through a visual interface that rises from the surface. 
DUls appear on objects rather than being the object 
itself or part of it, as in the case of TUIs [10]. As GUls, 
they display the information on screens, but not the 
traditional ones. The interface is integrated into the 
same material that composes the devices, and the 
feedback is given by the very skin of the object that 
changes depending on the information they receive or 
to explain the result of an operation. The interface is 
embedded in the material, and what changes is either 
its chemical or physical properties. Designing interfaces 
that draw themselves implies to alter and modify users' 
perception of objects. In so doing, these interfaces 
advance a reasoning that grounds on hacking objects' 


meanings [23]. 


2. Process and Methods 


The research is conducted by Thingk, a spin-off of the 
Politecnico di Milano that experiments with interactive 
technologies, designing products that disguise 
themselves. It digs into apparently simple and minimal 
artifacts (form) that are technologically enriched 
(function), following the slogan: "objects of daily use 
with superpowers" (thingk.design). Thingk takes 
advantage of the principles of digital transformation 

[24] while pursuing technological hybridization, blending 
analogue properties and digital immateriality. This 
approach results into objects with invisible technology 
and essential aesthetics, that are augmented with 
unexpected properties. By redesigning their appearance 
and functions, hence affordances and interactions, 

the meaning of the artefacts gets affected. Our study 
capitalizes on how different skins activate different 
functions, changing how they are interpreted and 


approached by user [25, 26]. 


Experimenting on how to use and innovate with DUls, 
we apply an iterative design process (fig. 1) where 
multiple approaches and methodologies are combined 
to answer different needs [21]. The overall process 
consists of two main macro-phases. The first follows a 
design-driven innovation concept [27] that is applied to 
the research and design phases, when prosumers are 


involved in co-design sessions with early prototypes 
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to provide insights. The latter revolves around user 
testing and data analysis and regards development and 


production. 


The Research phase focused on deepening our 
knowledge about contexts where several elements 

are intertwined, interacting among them and with 

the surrounding environment. We analysed lifestyle 
and technological trends (state of the art) to point 

out potentialities and constraints of the context of 
reference, especially breakthroughs in the contiguous 
areas of interaction design, electronics, and material 
research. Acknowledging the existence of interfaces 
that emerge from the surface as Mui’s Calm Design, and 
our own Slab!, we are also aware of their limits, as their 
being static and localized. This suggested the concept 
of a multilayered interface in which layers are activated 
according to the function required. We tapped into 
extending from multiple functions to multiple language 
and meanings that coexist under the skin of the same 


object. 


We are currently running the Design phase, with 
prototypes of use expected by the end of this year. 
Informed by the research conducted, we decided to 
completely hide the interface until the object detects 
motion, an interaction, or a connection with other 
objects. At this point, the interface appears. This phase 
involves lead users to verify their response towards 
the product/interfaces designed, gathering data from 


survey and qualitative enquiry (from rapid ethnography 
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and participant observation to interviews), but also 
focus groups and workshops where sketches, renders 
and use cases are leveraged to stimulate discussion 

and validate our design choices. The Development and 
Production phases are expected to respectively start 

in one and two years. From an engineering and design 
perspective, the challenge is developing objects that 
reconfigure their own function according to the surface 
that they show, in a logic of quick adaptability to the 
context of use. We are currently conducting studies 

on possible technologies (electrochromic displays) and 
application (from smart product to smart buildings), 

as part of DecoChrom, a Horizon 2020 Project that 

will end in 2021. During the project we will verify 

DUls flexibility through three different applications on 
everyday objects. Working prototypes will be tested 
with the DecoChrom partners and a with community of 
prosumers, conducting user analysis to identify possible 


improvements. 


3. Results 


In conceptualizing the meaning of design, Krippendorff 
[25] described it as the attribution of meaning to 
things. Artifacts should, in fact, communicate their 
function through their aesthetic, which should 
introduce users to their correct functioning [28]. 

As theorized by Tidwell [29] the interface has an 
operational function that triggers the dialogue with the 
user. Therefore, redesigning the meaning of objects 

is a practice of a certain importance, which becomes 


paramount if we consider that Thingk reframes the 


form-function relationship of products, pursuing an 
innovation inspired by design, while recognizing users' 
needs [26], and the current growing technological 
opportunities [27]. When objects change their meaning, 
their language becomes anything but obvious [30]. 
Examples are those interactive artefacts that rely on 
integrated technological component (loT) to become 
extended systems that dialogue with applications 

and other objects. Starting from such assumptions, 

we conceptualized a model of variable and situated 
interfaces that reacts not just to the interactions 
triggered by users, but also to the surrounding 
environment (context of use). Recognising the 
limitations and constraints of both TUIs [14] and NUIs 
[8], and being aware of the invisibility dilemma [31] that 
comes along, we conceived a typology of dynamic 
interface that expresses itself and communicates to 

the user what its form hides (affordance). In an attempt 
to integrate smart functionality into everyday objects 
without falling into the aforementioned dilemma of 
choosing between minimizing disturbance from the 
main task and adding value including explicit interaction, 


there is a tendency to hide object's intelligence. 


However, the consequence of keeping/making invisible 
to the user such an increase of functionality results into 
a dichotomy between aesthetics and functions. Making 
functionalities recognisable allows users to identify 
such augmented artifacts as smart and hence use them 
appropriately. In parallel, the control dilemma [32] 
exposes a crucial point when dealing with smart object 
automation. The attention lies in designing objects that 
act and react autonomously to certain inputs, hiding 
the ongoing complexity to the user without lessening 
his/her perception of applying control. On this regard, 
significant is the study on the use of the remote control 
Nest Learning Thermostat, where the users' interviewed 
stated they were not able to fully understand the object 
learning process of their habits, hence they were not 
able to utterly rely on how the smart home automation 
device was self-setting certain functionality. The control 
dilemma requires to pay attention to the user, who 
wants to know what is going on; this insight is on the 


ground of DUls concept. 


3.1 Ecosystemic (Situated) User Experiences 


Focusing on functionality, we often face the problem 


of how much information an interface should return 

in order to perform each task. The advantage lies on 
using a different set of variable commands that allows to 
isolate and “situate”, or contextualize, a device's main 
functions according to its actual use case. Employing 
DUls allows indeed to define situated user experiences 
that are variable in time and space since they are closely 
related to the context in which they are immersed. 
Moreover, these interfaces can be modified and 
updated over time, potentially changing the product 
functionality: the object doesn't change, it updates its 
functions. In this perspective, an object or device can 
change the interface depending on, for example, its own 
positioning, and inclination, or according to the location 


of the user. 


So far, we identified three possible use cases: 

* Movement: Response to being relocated within an 
environment 

* Reaction: Response to environmental changes 
(presence of people, physical variations, and other 
modification in or of the context) 

* Dialogue: Functional variation based on the 
relationship with other connected objects present in 


the surrounding space 


Below we present existing case studies as pretexts to 
explain possible use cases of DUIs. Describing their 
functioning we explain how DUls could be applied 

to perform similar functions, making the interaction 
more direct, complete and immediate. In the first 

use case, Movement, the interface responds to the 
change of position and orientation, revealing itself and 
displaying different functions. Microsoft’ Surface Dial is 
a controller that interacts with different programs and 
digital elements, serving specific functions according 

to how it is activated and manipulated. However, the 
function of the interface is delegated to an external 
screen, where the information is displayed, while the 
product itself does not provide any visual feedback 

on its surface. In this case study, the potential use of 
DUls lies in overlapping object and interface, so that 
functions are communicated directly on the product. 
In the second use case, Reaction, the interface reacts to 
proximity or interaction with external elements, which 
can be as much the user as the environment with its 
variables. Lapka's Environment consists of a series of four 


environmental sensors embedded into as much small 
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devices, translating complex data into user-friendly 
visualizations displayed on the smartphone. Once 
again, the function of showing data is entrusted to an 
external element: the smartphone display. Mounting 
DUls could allow to provide immediate feedback on 
the product itself. In the last use case, Dialogue, the 
interface relates to the surrounding smart objects. 
Ikea's Tradfri remote control connects to different 
lamps working by proximity, managing on and off, dim, 
and color temperature. The remote control recognizes 
and regulate bulbs within a range of 10m, but does not 
provide information or feedback. In this case, applying 
DUlIs would allow to show the object to which the 
remote control is connected, clarifying the functions 
available. In addition, in the hypothesis of multiple 
smart objects in the surrounding environment that 
could be managed by a single controller, DUIs could 
change the controller interface depending on the 
object ruled and the function that can be performed. 
Changing its communicative skin, the object performs 
tailored functions depending on the interpretation of 
the surrounding variables. The research phase brought 
to think to functional layers that overlap, populating a 
single object with multiple capabilities. The main result 
of our study is designing the transition among various 
commands and as many functions that coexist in the 
same object. The output of our investigation is a surface 
capable of smoothly changing and reacting during the 


user experience. 


To provide a sharp idea of how they work, fig. 2 


presents a use case of DUIs applied on a smart object 
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acting as a controller. The object can be exemplified as a 
cylinder embedding smart sensors that can perform the 
regulation of values as temperature, volume, and so on, 
and that is provided with a silent interface. Such object 
contains multiple layers, each with its own function that 
gets activated according to the previously described use 
cases. The object becomes a repository of information 
that is going be communicated once used. For example, 
one layer could show how to regulate the temperature 
through rotation and selection; another one could 
display a linear scale to operate on the intensity of 

light, using rotation to dimmer; then, the third layer 
could allow to play/manage music or movies through 
interactions as selection, rotation and swipe. This 
design scenario is an example of possible use cases, 

as employing DUls to increase the variables at play, 
introducing an auto-nomatic dimension [33] that enhances 
the qualities of the elements involved. Attributing a 
performing ability to the interface, its visual elements 
gain the ability to instantly act and react to manipulation 
[34]. These feature and dynamicity require to design 
considering the relation between the object and the 
environment both in the form of positional/static 
identities and as variable elements that are fluid and 


responsive in their composition. 


4. Discussion. A Discourse on Possibilities 
and Constraints 


The use cases described can be developed harnessing 
various technology. The ongoing tendency to 


dematerialise objects' physical interfaces frequently 


led to delegate the operational functions of an object 
to digital screens. Indeed, most of the contemporary 
smart products uses embedded displays or transfer 
data to other devices with screens (mobile and tablet). 
Such displays, acting as communication gateway, are in 
charge of returning information. That said, the current 
technological state of the art witnesses the appearance 
of smart materials able to reconfigure their shape, 
becoming potential drivers of information. However, we 
are still far from the concrete possibility of a material 
so ductile to display a continuous and updated flow 

of digital data, through a modification of its intrinsic 
physical properties. At this state of technology, DUIs 
are affected by this restriction, and they present a 
further limitation: since they allow an object to embed 
multiple functionalities, they do not make any function 
manifest by default. The absence of the interface, when 
none of the layers is active, as well as the presence of 

a variable interface concealed until the moment of use, 
affects the way in which the object communicates and is 


interpreted by the user. 


In addition, their being entirely based on visual elements 
makes them unsuitable for being used by people 

with visual disabilities. By contrast, the information 
behind these visual components could be translated 
or implemented by combining them with additional 
sensory stimuli. For example, dealing with TUls, Ishii 
recommends the use of more “malleable” forms of 
feedback, such as audio and video, that complement, 
support and complete interfaces. Our reasoning on 
DUls possible implementations goes in a direction 
similar to what Ishii [10] identified as Double Interaction 
Loop, namely a cycle that starts with a first and 
immediate level of interaction and haptic feedback 
provided in consequence of touching, grasping and 
manipulating a physical object, followed by intangible 
feedback (as audio) and possible physical modification of 
the object, reflecting a change of the digital data. 

A possible field of application of DUIs is between 
analog (aesthetics and materials) and digital (functions 
and transitions). Several technologies can be used 

to build such interfaces. Particularly, we investigate 
those materials that change the state of their chemical 
(electrochromic) or physical (e-ink) properties. Both 
technologies can be integrated into surfaces made of 


natural materials—as wood, marble, metal—without 


contaminating their appearance and tactile properties. 
On the one hand, aiming at the e-ink, as in the case 

of the Yotaphone, ensures excellent graphic resolution 
but has a color range limited to black and white. 

On the other hand, electrochromic screens have 

the advantage of being the result of a monochrome 
screen printing technique on transparent surfaces, 
allowing them to be positioned above the material of 
the object itself. However, since electrochromic is the 
technology subject of the research of the European 
project DecoChrom, it is currently the basis on which 
we are conducting the experimentation. An advantage 
of choosing this not-new technology is that the 
electrochromic absorbs energy only when transitioning 
from a state of transparency to that of opacity; once a 
state is reached, there is no consumption related to its 


maintenance. 


Designing interfaces we follow a model of 
experimentation closely connected to the field of loT 
where the relationship between objects, technology 
and context is at the heart of the project. We privilege 
a design-driven innovation method that takes into 
account the needs of the user. Hence, the innovation 
is oriented to the relationship between function and 
aesthetics, and it relates to the context of use. One 

of the results achieved is to preserve the aesthetics 

of products that historically did not mount interfaces, 
inserting an interactive layer capable of manifesting 
itself and disappearing when needed. The implications 
of integrating physical elements with computational 
processes expand the design possibilities in terms of 
choosing materials that become reactive, dynamic and 
able to shape, transform and inform [13]. We consider 
this reasoning significant to various disciplines, from 
design to HCI, as it explores how interfaces can exploit 
smart technologies to modify themselves while mixing 


digital and physical. 


As said, DUIs are currently in the prototyping phase, 

as user research will take place in 2020. The main 

issue that will be the subject of our upcoming enquire 
concerns how users will react in front of interfaces that 
design themselves when necessary, namely changing 
according to their being placed in space, the dialogue 
with other smart objects of the digital ecosystem, or 


the interaction with the user. In doing so, it is central to 
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understand the role of aesthetics in dynamic digital- 

physical ecosystems. Addressing the design of interfaces 

for distributed, hyperconnected, and complex smart 

ecosystem requires to tap into the implications of 

having multiple meanings coexisting into the same 

object; meanings that are not persistently displayed, but 

emerge from the surface when needed. In consequence, 

it is crucial to enquire the way in which the users 

will interpret and interact with objects that contain 

interfaces that are simultaneously: 

* Hidden, since they are not always displayed 

* Variable, since they reconfigure themselves according 
to the environment 

* Updatable, since they are designed to last more time 


than the lifespan of the object itself. 


To conclude, DUls take advantage of how micro- 
processors and sensors can provide an ecosystem of 
smart objects with awareness of their surrounding, 
exploiting their ability to dialogue with users, other 
objects and the environment. These interfaces can 
disappear, hence they can be used on objects and 
products that maintain their overall aesthetics while 
being augmented. The interface is no more influencing 
or contaminating the look-and-feel of a product. That 
said, future developments concern the enquiry on users 
to understand how such interfaces have implications in 
terms of experience; in parallel, possible advances of 
electrochromic technology, as well as the application 
of other technologies, could positively impact on the 


project. 
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Individual Mid-Air Gesture 

Sets Informed by Conceptual 
Metaphors: A Case Study on How 
Users Generate Mid-Air Gesture 
Sets to Control Video Streaming 


Abstract 


Recent research on mid-air gesture interaction for 

TV control aimed to standardize them. To this end, 
researchers developed a design approach that relies 
on the agreement rates among the elicited end-user 
gestures. Contrasting with the agreement based 
approach; a recent study have shown that the most 
common mid-air gestures might not be the most 
favored ones. In addition to this, researchers claimed 
that the agreement studies ignore users’ cultural and 
conceptual bias. Thus, it can be postulated that the 
mid-air gesture interaction research can benefit from a 
qualitative analysis of the users’ mid-gesture set design 
processes. Towards this end, this study investigated 
users’ task conceptualizations and mental models. For 
this purpose, a mid-air gesture-based video streaming 
experiment was simulated with 10 participants, 4 
females and 6 males. Through the lens of Conceptual 
Metaphor Theory, the study investigated the similarities 
between the participants’ conceptual representations. 
The study findings demonstrated that the participants’ 
conceptualizations had clear references to their bodies 
and prior physical experiences with the objects, which 
reflected as linguistic representations of orientational 
and ontological metaphors in participants’ explanations. 
Further findings of the study addressed intersections 


between participants’ mental models. 
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1. Introduction 


TV and streaming technologies have advanced 
significantly in the last 10 years. With the rise 

of interconnectivity, streaming platforms have 
revolutionized the TV experience. The number of 
streaming platform use per household has surpassed 
that of cable TV use in some countries [1]. The 
evolution in content accessibility has reinforced 

the idea of redesigning our interaction with TV [2]. 
Research on novel TV control systems projects a future 
interaction experience that surpasses the limits of 
graphical interfaces. Research in the Human-Computer 
Interaction field has highlighted that mid-air gesture 
interaction may deliver the desired "natural" interaction 
experience. However, researchers have found that 

the standardization of the mid-air gesture interaction 
was problematic. In pursuit of a standardized mid-air 
gesture vocabulary, several elicitation studies were 
conducted. These studies aimed to find the most 
recurring mid-air gestures to design the gesture sets. In 
this way, the produced mid-air gestures could become 


the archetypes of their category. However, in a recent 


study, researchers postulated that the agreement- 
based approach ignores the users' conceptual, cultural 
and physical bias [3]. Thus, presented agreement 
calculations were inflated. Moreover, another group 

of researchers found that the mid-air gesture sets 

can vary across cultures and that the most common 
gestures might not provide the best design solutions 
[4]. Thus, it can be speculated that the most suitable 
design approach for mid-air gesture set has not yet been 
formulated. The underlying reason for this problem may 
be the prevailing gesture consensus approach in such 
studies because the approach neglected the importance 
of visual references and assumed that the human 
interpretation of the outputs and verbal commands 
could be standardized. Moreover, the method also 
postulated that the users could always design coherent, 
engaging mid-air gesture sets in a limited time period. 
Alternatively, in a recent research, a turn towards a 
human-centered design approach has been suggested 
[4]. The study evaluated users' appreciation for all of 
the mid-air gestures regardless of the agreement rates 
and found that the most agreed gestures may not be 
the most appreciated ones. In line with this imperative 
debate on the design approach, this study aimed to get 
a better understanding of the users' mid-air gesture 

set design processes. With this aim, a computer- 
operated streaming control experience was conducted 
with 10 participants. Throughout the experiment, the 
participants controlled a streaming activity with their 
mid-air gestures, and after demonstrating each gesture, 
the participants explained their thought processes. 
Later in the study, with the guidance of Conceptual 
Metaphor Theory (CMT), the users' conceptual 
representations for the given tasks, semantic similarities 
between these conceptual representations, the users' 
task grouping behaviors, and intersecting mental models 
were investigated. By doing so, the study evaluated the 
possibility of employing a mental-model-based design 
approach. The contributions of this study to the design 
field are (1) presentation of the collected mid-air 
gesture sets for streaming control, (2) examination 

of the conceptual representations of the tasks, (3) 
presentation of shared conceptual representations of 
the tasks and participants' intersecting mental models 
and (4) suggestions for the design of mid-air-gesture- 


based streaming experience. 


2. Background Work 


2.1 Conceptual Metaphor Theory (CMT) 


Metaphors are commonly used to explain one thing in 
terms of another. In 1980, Lakoff and Johnson proposed 
through their CMT that humans' conceptual thought 
processes are comprised of metaphors [5]. Metaphors 
reflect the diversity of our cultural, physical and 
cognitive experiences. Based on how they are formed, 
Lakoff and Johnson categorized conceptual metaphors 


as structural, ontological and orientational. 


A structural metaphor enables us to define one concept 
in terms of another concept. For instance, "Argument 

is War" metaphor uses "Argument" and "War" as 
concepts. To understand which aspect of the "War" 

is assigned to the "Argument" concept, we employ 

the most defined qualities of the “War” concept. 
Ontological metaphors differ from the structural 
metaphors in the nature of their source domains. The 
source domains of ontological metaphors are concrete 
objects, tangibles, and entities. For instance, “The mind 
is a machine" metaphor uses the "machine" as its source 
domain and assigns its qualities to the "mind" concept. 
In this way, the mind can be explained with the qualities 


attributed to a machine. 


Structural and ontological metaphors show similarities 
in the way they are formed. Both of them explain 
unfamiliar concepts in terms of other familiar domains. 
However, in some cases, we explain the concepts 

in terms of relationships. Orientational metaphors 

are formed by relying on these relationships. They 
define a whole system of concepts in terms of spatial 
relationships, such as up-down, deep-shallow, or on-off. 
For instance, "having control over a situation" is based 


on the conception that "force is up". 


In summary, CMT advocates that direct, embodied 
physical experiences can shape our conceptual thinking 
[6]. As mid-air gesture interaction relies on bodily 
communication, herein it is postulated that CMT can 
guide us to decipher the source of conceptualizations 
represented by mid-air gestures. Although Conceptual 
Metaphor Theory stands on linguistic evidence, research 
on cognitive science demonstrated that both linguistic 
and bodily expressions are resourceful in explaining 


our conceptualization [7]. Hence, in this study, the 
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analysis of users' self-reported statements and gestural 
expressions were employed to unfold the conceptual 
representations in the participants' mid-air gestures. 
The analysis of the collected materials benefited from 
CMT in classifying the participants' recurring conceptual 
representations and gesture grouping behaviors. 
Classified conceptual representations depicted the type 
of sources that users consulted while forming their mid- 


air gestures. 


2.2 Gesture Taxonomy and Agreement Based 
Approach 


Previous research on mid-air gesture interaction for 

TV control systems has focused on eliciting end-users' 
gestures and proposing methods to design intuitive 
gesture sets. In the last 5 years, 74.5 % of studies have 
employed Wobbrock et al.'s "guessability method" to 
design the most intuitive gesture sets [8]. The remaining 
studies have used "choice-based elicitation", "Nielsen's 
intuitive and ergonomic model", "a combination 

of Wobbrock's and Nielsen's models" and other 
methods [8]. Due to the prevalence of Wobbrock et 
al.'s classification model and guessability method, this 


section explains them in detail. 


According to Wobbrock et al., gestures can be classified 
based on four dimensions: their form, nature, flow, and 
their binding elements. These dimensions are used in 
defining a gesture's movement such as static, dynamic 
and one point; a gesture's relationship with the object 
and space around it, such as object dependent, world 
dependent, world independent and mixed dependencies; 
the continuity of the gesture to be recognized, such 

as continuous or discrete; and the semantics of the 
gestures, such as symbolic, physical, metaphoric and 
abstract [9]. Symbolic gestures refer to the symbolic 
representations, for e.g., the well-known "OK" and 
"STOP" poses. Physical gestures are demonstrations 

of the actual physical interaction with the objects 

on screen/in the air [9]. Metaphorical gestures rely 

on semantic associations between the gestures and 
delivered functions. Abstract gestures are the remaining 
gestures that do not convey any association with the 
referred function [8]. Often, based on this taxonomy, 
researchers employed end-user elicitation studies and 
guessability method to standardize mid-air gesture sets. 


In a typical elicitation study, participants are directed 
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to produce mid-air gestures for certain referents 

and depending on the employed study method, the 
most recurring or correctly guessed mid-air gestures 
become the representative of the commands (referent). 
However, recent research has demonstrated that 
agreement rates across individuals' mid-air gestures 
could be quite low in studied samples; moreover, 
mid-air gesture sets could vary across cultures 

[4]. In addition to that, the most common mid-air 
gestures might not indicate that it would be the most 
appreciated one [4]. Therefore, it is not clear how 
mid-air gesture sets should be designed. In this sense, 
users' conceptualization processes can help formulate a 
better design approach. Towards this end, the present 
study aimed to understand the type of conceptual 
representations users deployed to form their mid-air 


gestures. 


2.3 Mid-Air Gesture Interaction for Interactive 
TV Control 


First introduced by Freiman and Weisman, dynamic 
hand gestures are now being proposed as a novel way 
of interaction with TV [10]. Since the emergence of 
dynamic hand gestures, several studies investigated the 
possibility of redesigning interactive TV experience 
with mid-air gesture interaction. For instance, studies 
by Zaiti et al. and Wu et al. investigated users' mid-air 
gesture preferences for TV functions, and Vavatu et al. 
compared user agreement rates between hands-free 
interaction and augmented remote control interaction 
[2], [11], [12]. Further research on mid-air gesture- 
based TV control examined the possibilities of hands- 
free text entry for Interactive TVs and preferences 

for mid-air gesture control among older adults' [13], 
[14]. Research on mid-air gesture interaction for 

TV frequently progressed through end-user gesture 
elicitation and choice-based elicitation studies. Research 
in this field constantly aimed to standardize mid-air 
gesture-based TV interaction often relying on the 


consensus of the collected mid-air gestures. 


3. Experiment 


3.1 Participants 


Ten participants (F: 4, M: 6, Median: 31), of ages ranging 
from 29 to 33 years, volunteered for the present study. 


Their professions varied: three aerospace engineers, 
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of the Experiment 
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three mechanical engineers, one system engineer, 

one mathematics teacher, and two psychologists. The 
main criteria for the selection of this participant group 
were that (i) the participants reported prior on-screen 


gesture experience ranging between six to eight years; 


were instructed to interact with hypothetically 
autonomous subjects [15]. Throughout the experiment, 
the wizard sat behind the participants so that she 
would not influence the participants' gesture design 
process, as shown in Fig.1. Participants performed the 
announced tasks using mid-air gestures, and after the 
announcements of the tasks, the wizard simulated the 
effects. After each gesture, the participants explained 
what inspired their gestures. The wizard operated 

the "YouTube" video streaming platform in full-screen 
mode in this case. The choice of the video platform 
was based on the participants' familiarity with it. In this 
context, the participants generated mid-air gestures for 


the announced tasks in the following order: "turn on 


the laptop", "turn on the speaker", "connect/pair the 
devices", "play", "pause", "adjust the volume", "adjust 
the speed", "adjust the video quality", "subtitles on", 


"subtitles off" and "watch later". With the consent of 


Fig. 2. Gestures of 
the study participants 
for the "Turn On" 


Command and the 


metaphorical references 


in gestures 


Opening a book 
P1 


(ii) frequent use of online video streaming services; the participants, the wizard recorded the whole process 


(iii) they had no prior experience with mid-air gesture using a MacBook Pro in-built camera. 


control systems; and (iv) they were motivated to 


perform the necessary tasks. 


4. Study Findings 


3.3 Procedure 4.1 Conceptual Representations of the 


Commands 
Prior to the study, the participants were briefed about 


the mid-air gesture interaction. The study took place in Ahissecunp presente the analysis ofthe conceptual 


a home environment, which the participants visited 1 to representations of the commands (referents) reflected 


3 times per week. The participants’ familiarity with the in the participants’ mid-air gestures. 
household could assure that the experiment was close 


€€ » s or . . 
to a real experience. On their arrival, the participants Turn'On' s Firstly, the participants performed mid-air 


were informed once more that they could use either gestures fo Bie mp" Dum en-somimand tothelaptop 


a single hand or both hands to perform the mid-air computer lo Turion fe laptop, Fardcipant-2(2) 


" * " 
gestures. The test apparatuses were a digital projector, demonstrated a "greeting" gesture, P4 and P8 used the 


a projection screen (2.6 x 2 meters), a speaker and transition of their palms from a closed state to open 


a laptop, the use of which the participants were state, and the gesture was supported by the rise of 


already familiar with. The study was performed using their arm along a diagonal axis. P4 and P8 reported that 


the Wizard of Oz method, wherein the participants they used visible; Up:and ready- palmi! to represent 


“being on”. P6, P7, and P10 also demonstrated the "rise 


GESTURE FOR “TURN ON” 
COMMAND - SPEAKER 


Gestures of the participants, who preffered 
to use different gestures to activate the speaker 


GESTURES FOR “TURN ON” COMMAND - COMPUTER 


nO 


Pressing on an 


Stand Up / Rise 
Imaginary Button 
P9 


Greating 
P2 P6, P7, P10 


Sliding the black Finger Snapping Gazing In Air 
Screen Making Sound Looking For 


P5 P2, P4 Sound Waves 
P8 


Swirling on an 
imaginary Button 
P3 


Commanding the 
device to become 


up 
P4, P8 


Design and Semantics of Form and Movement 125 


GESTURES FOR THE “PAIR” COMMAND 


Merging the strings Quick Transfer 

of the devices in air 

Communication is Communication is P3 
meeting meeting 


P1 P2 


Clapping, bringing 
devices together 


& stand up" movements in air with their palms facing 
upwards, rising along the vertical axis. Conceptual 
representations of the tasks in 6 participants' mid-air 
gestures correlated with the orientational metaphors 
defined by Lakoff as "conscious is up" and "force is 
up" [5]. Of the remaining four participants, P3 and P9 
delivered the command by interacting with imaginary 
buttons. P1 demonstrated two open facing palms 
extending along the horizontal axis and reported that 
the movement depicted "opening a book". P5 swiped 
her right palm starting from her left to the right along 
the horizontal axis. The gestures are summarized in 
fig.2. Later, the participants were also asked to turn 
on the Bluetooth speaker with mid-air gestures. P2, 
P4, and P8 mentioned that the speaker would require 
different mid-air gestures since its predominant purpose 
was playing music, while the remaining participants 
maintained the same mid-air gestures for both devices. 
P2 and P4 snapped their thumb and middle fingers, 
given that since ancient times, snapping fingers had 
represented tracing the rhythm [16]. Thus, it can be 
speculated that the participants referred to the speaker 
as a company, “who” is capable of responding to them. 
P8 depicted an open palm slowly moving along the 
peripheral axis from left to right in front of her upper 
torso and explained that the movement represented 


"gathering sound waves". 


“Pair”. As the third task, the participants were 

asked to use mid-air gestures to "pair" the laptop and 
Bluetooth speaker. The participants were informed that 
the devices' Bluetooth function was on in its default 
mode. As shown in Fig.3, to connect/pair the devices, 
P6 & P7 traced a half circle between the objects using 
their index fingers and P4 & P9 traced a half circle 
between the objects with their palms. P3 and P5 
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Transferring imaginary Transferring with index Drawing quick paths 
finger on a half circle between devices 
Commuication is sending Commuication is sending 


P6, P7 P10 


Finger Snapping 
Instantly Bringing Devices 
Together 
Communication is 
meeting 
P8 


Commuication is sending 
P4, P5, P9 


swiped a single open palm along the horizontal axis Fia d, Gosures efie 


and P10 used a single palm facing downwards to wave study parici for 


between two imaginary points. These participants the "Pair" Command 


self-reported explanations indicated that the gestures anddhenpetashodes 


represented a hypothetical entity is sent through a veles n petur 
half-circle or a horizontal path between the devices. 

Conceptual representations of the task correlated with 

“communication is sending” metaphor. The remaining 

participants executed clapping, finger snapping and 

gradually approaching palm gestures. These participants 

mentioned that the movement signified "the union of 


the devices". 


“Play”. To play the video, of the ten, six participants 
directly pointed to a random spot on the screen, and of 
them, only P1 avoided using the index finger on its own 
for the pointing gesture. Instead, he used his index and 
middle fingers firmly held together and pointed at the 
play button as if he was firing a gun. P7 used a finger- 
snapping gesture, P5 and P8 used the rising opening 
palm gesture, P2 moved his finger up and P10 swiped 
his hand to the right. In general, the participants used 
pointing gestures and performed the gesture in a quick 
discrete movement. However, P2, P5 and P8’s gestures 


correlated with Lakoff’s “active is up” metaphor. 


“Pause”. Of the ten participants, five used the pointing 
gesture to give the “Pause” command. One participant 
closed her palm to a shrunk position and one participant 
used the finger snapping gesture. Two participants 

used the static open palm pose and one participant 
moved down his finger. In general, the participants 
either repeated the same gestures they used to give the 
"Play" command or demonstrated the reverse motion. 
Additionally, two participants used the static "Pause" 


pose. 
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Fig. 4. Gestures of 
the study participants 
for the “Play” & 
“Pause” commands 
and the metaphorical 
references in 


gestures 
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Volume Adjustment Control. All of the participants 
demonstrated their gestures along the vertical axis. 
They either moved an index finger or a single palm 
along this axis. Only P1 moved his finger around his 

ear to adjust the volume; however, even he used the 
vertical axis for adjustment. The participants reasoned 
that spatial highness represents more. Previously, in 
embodied music cognition, it is reasoned that vertical 
representation of the sound amplitude aligns with 
"greater is higher" metaphor [17]. The findings of this 
study correlated with this claim, as all of the participants 


used the vertical axis for volume adjustment. 


Speed Adjustment Control. The experiment 
continued with speed adjustment command. P3 
relied on his prior on-screen gesture experience, 
and consequently, used the double-tap gesture. P9 
directly pointed his fingers to the screen. P7 used 
finger snapping, P2 tapped on his face and P4 used 
the swiping gesture. P5 tapped and bounced her 
hand periodically on the horizontal axis while P1, P6, 
P8, and P10 illustrated a periodic cyclic motion. All 
except P3 and P9 adjusted the speed of their hand 
motions to demonstrate the speed required. Except 


for the pointing gestures, the participants’ mid-air 


Shrunk ,Hidden Palm 
P1 


Getting into the 
down position 
Active is Up 
P2 


Stop Sign 
P5, P10 


gestures were diverse, yet coherent. All of the gestures 
demonstrated a "periodic" motion either along a 
linear or cyclic path. The participants associated the 


» « 


command with the motions of "bouncing", "swirling", 
"swiping" and "rolling". Some of the participants who 
demonstrated a cyclic motion reported that cassette 
tapes inspired their mid-air gestures. To give the reverse 


commands, the participants just reversed the directions. 


Video Quality Adjustment. Five participants 
pointed to the screen to adjust the video quality. P6 
moved her right open palm along the vertical axis 

as she previously used her left hand for the sound 
adjustment. The remaining participants demonstrated 
varying mid-air gestures. P2 moved his thumb in “OK” 
position along the vertical axis. P5 demonstrated a 
gesture that resembled drawing a "V" shape in the air 
and she reported that the gesture is inspired by the 
"sunrise" metaphor. P7 tapped on his eye to adjust the 
video quality and P8 rubbed her palms on a hypothetical 
surface in air. P7 reported that he used pointing the eye 
as an indicator of better vision and P8 reported that 
rubbing surface refers to clearer vision. These mid- 

air gestures were semantically associated with vision. 


Vision is often associated with brightness and clearance, 
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and the references were coherent with the commands 
[18]. Interestingly, the study participants had difficulty in 
finding gestures to adjust the video quality. All of them 


asked for extra time to find a mid-air gesture. 


*Watch Later". To provide the "watch later" 
command, two participants directly pointed to the 
graphical interface and swirled their index fingers in 
the clockwise direction. Two participants pointed to 
their skulls with their index fingers. These participants 
reasoned that pointing to the skull meant storing the 
information in the brain. Four participants grasped 
the content with a single hand and placed it in an 
imaginary container. One participant reported that she 
perceived the content as a shopping item. Conceptual 
representations of the command indicated that the 
content is often perceived as a physical entity that can 


be grasped and stored in a container. 
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“Turn On & Off the Subtitles”. To turn on the 
subtitles, P1 and P4 used pointing gestures. P3, P7, P8, 
P9, and P10 swiped their right hand or fingers along the 
horizontal axis. The position of their hands emphasized 
the location where the subtitles were often placed 

on video players. Three participants demonstrated 
gestures that did not have any reference to the graphical 
interface. P2 revolved his right palm from downward 

to upward direction and rationalized that "facing up" 
means "being on". P6 used typing gesture with both 
hands moving along the vertical axis to turn on & off the 
subtitles. P5 grasped an imaginary entity with her hand. 
Subtitle control gestures were often physical. However, 
the representation of "active is up" metaphor was 
visible in P2 and P6's gestures. Additionally, P5's gesture 


represented "understanding is grasping" metaphor. 
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4.2 Shared Conceptual Representations of the 
Commands 


The participants' shared conceptualizations of the tasks 


are summarized as follows: 


Turn on the Laptop. "Force is up" metaphor. 

Turn on the Speaker. The personification of the 
speaker. The participants referred to the speaker as "a 
person/entity that can respond to sound". 

Pair the Devices. "Communication is sending" and 
"communication is meeting" metaphors. 

Play & Pause. "Active is up" metaphor. 

Adjust Volume. "Higher is greater" metaphor. 
Adjust Video Speed. Representation of the video as 
"a rolling circular entity" and "speed as rhythm". 
Adjust Video Quality. The shared conceptualization 
correlated with "higher is greater" metaphor, however, 
the participants also employed interesting metaphors 
such as "video as a surface", "visual quality as sunrise" 
and "the eye as a button". P6 used "Greater is higher" 
metaphor to link the task with the volume adjustment 
task. 

Turn on the Subtitles. The participants agreed on 
pointing towards the typical location of subtitles on 
graphical interfaces. 

Watch Later. “Video as a graspable entity" and “brain 


as a container" metaphors. 


4.3 Intersecting Mental Models 


Pointing Gestures for Relatively Complex 
Commands. According to the study results, the 
participants used pointing gestures when they had 
difficulty in finding suitable physical demonstrations. 
A study by Mauney et al. found similar results [19]. 


However, the "Play" command can be an exception to 


location of subtitles on 
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Typing and 
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Active is up 
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Pointing to the 


this generalization since the participants used pointing 


gesture in a distinguishable way. 


Reverse Gestures for the Dichotomous 
Referents. Previous studies have shown that the users 
often preferred reversible gestures for the dichotomous 
referents and the findings of this study supported this 
inference [2], [9]. This behavior often surfaced in the 


demonstrations of physical experiences. 


Same Gestures for the Dichotomous Referents. 
Study findings have shown that some of the participants 
employed the use of same gestures for dichotomous 
referents, such as for "Play / Pause" and "Subtitles on/ 
off" commands. The participants who employed this 
model associated these commands with clicking the 
buttons. Our findings support the findings of a previous 
study by Kühnel, C. et al. [20]. 


Redefining the Same Gesture with the “Stop” 
Pose for Dichotomous Referents. Even though the 
participants were encouraged to use a single gesture, 
three participants used the static "stop" pose and "OK" 


pose at the end of certain gestures. 


Limited Hand/Finger Movements. Two participants 
tried to limit their hand/finger use. P9 used her right 
index finger for most of the commands and P2 placed 


his fingers on the armrest, which restricted his motions. 


Transferring On-Screen Gestures to Mid-Air 
Gesture Interaction. Our findings have shown that a 
few participants used on-screen gestures as their mid- 


air gestures. 
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5. Discussion 


This study aimed to understand how users generated 
their mid-air gestures through the analysis of their 
conceptual representations. The motive of the study 
was to take a step back from agreement calculations 
and understand the recurring patterns in users' 
mid-air gesture set design processes. Study findings 
showed on a conceptual level there were explicit 
similarities among the participants' mid-air gestures. 
Study findings indicated that the participants frequently 
represented orientational and ontological metaphors 
with their mid-air gestures. As aforementioned, 
orientational metaphors rely on spatial relations 
between the concepts. Ontological metaphors enable 
us to explain concepts in terms of tangibles. When 
controlling the streaming activity, the participants 
often described their conceptual representations 

as ontological and orientational metaphors. Thus, it 
can be speculated that the physical experiences with 
the objects and the embodied experiences played a 
central role in the design processes. However, when 
the commands did not referphysical experiences, e.g. 
video quality adjustment, the participants' conceptual 
representations often relied on structural metaphors. 
Based on these findings, some of the suggestions to the 


designers can be summarized as follows: 


Participants' Physical Experiences with Objects 
Inspires Mid-Air Gestures. The participants enjoyed 
depicting imaginary versions of physical experiences, 
such as commanding the computer to stand up, rolling 
the content as if they were rolling a physical wheel, 
grasping the content as if they were grasping an object 
and elevating the volume as if they were piling up things. 
Sometimes, they directly referred to their bodies. Thus, 
further explorations in mid-air gesture interaction 
should address physical interactions with different 


objects and how we refer to our bodies. 


Gesture Sets should be Recognizable when 
Performed in Different Volumes. Some of the 
participants purposefully used minimal mid-air gestures. 
However, older adults may not have the same physical 
abilities as young people do to perform exaggerated 
mid-air gestures. Thus, mid-air gesture sets should 


provide a ranging volume use. 
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On-screen Gestures may not Produce the 
Desired Engagement. Despite that the volume 
adjustment uses a horizontal bar on the mobile 
graphical interface, participants preferred vertical 
movement for volume adjustment. Thus, a simple 
transfer of on-screen gestures to in-air ones may not 


produce equal engagement in users. 


Conceptualizations are Grouped in Terms 

of Spatial, Temporal and Semantic Aspects. 

To evaluate the wholeness of the designed mid-air 
gesture sets, gestalt principles can be resourceful. 

The participants employed spatial, temporal and 
semantic references to group their mid-air gestures. 
For instance, some participants relied on "greater is 
higher" metaphor to group "adjust volume" and "adjust 
video quality" tasks. Both tasks had different semantic 
references, however, were grouped in terms of their 
adjustability along the vertical axis. In such cases, the 
proximity of the semantic relationships between the 
conceptual representations should be studied further. 
The proximity built instantaneously by the participants 


may be misleading in the design process. 


Enable Customization. The designed gesture 

sets should enable users to add mid-air gestures to 
the provided sets. In this way, the system becomes 
adaptable and users can feel in control of their gesture 


sets. 
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A Pedagogy for Noticing — 
Soma Literacy and the Designer 


Abstract 


The design fields strive for rich interactions fostered by 
ever deeper technologies yet continue to work without 
an overt conversation regarding the experiencing vessel, 
the sentient body. This body is evermore present in 
technological interactions, yet poorly understood when 
assumed only as one artifact among many. Before a 

true understanding of rich interactions can be realized, 
a turning of attention to the visceral variables of 
experience must first be achieved. The experiencing 
body, the body-in-motion, can be understood as the 
fundamental constant in all experience. It is through this 
bodily turn that an expanded palette of variables may be 
revealed to the design fields. Analyses and interventions 
through such a tactile and personal lens refocuses the 
conversation concerning interaction and experience. 
This paper describes a new course of study in Soma 
Literacy piloted at Carnegie Mellon School of Design 
and provides an introduction to Soma Literacy, a new 
framing and set of priorities applicable to designing 

for engagement of the felt-sense, providing both a 
timely provocation, and a much-needed practical aid to 


designers and educators. 
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1. Affinity to the Human Body 


How can a human body feel the digital? The recent tide 
of new technologies (in artificial intelligence, machine 
learning, nano tech, etc.) promises a world where the 
unimaginable becomes possible and problems previously 
unsurmountable are now within reach, yet how is 

the human to participate in such a world? As these 
technologies evolve, creating new cultural and social 
landscapes, it is prudent to take a moment to consider 
how it is that one might, in actuality, inhabit such a 
landscape. The technological world is certainly smarter, 
more interactive, and more connected, but what of the 
aesthetics of human interaction? What is the human 
experience of Al or machine learning? Is it possible to 


experience these things at all? 


| have investigated these ideas in a series of studies at 
the Carnegie Mellon School of Design, focusing on the 
human body as the center of all experience. It is only 
through a pushing-pulling, breathing—beating, shifting— 
leaning body where interactions become aesthetic and it 


is with this bias that | have framed the evolving research. 
“The knowable world...is the human body’s world, and only 
those elements that have some kind of affinity to the human 


body can enter it.” [1] 


Samuel Todes, writing in Body and World, lays out a 


deep case for the body as the one constant requirement 
for the creation of a knowable world. The integrated, 
moving body is the observer, translator, and actuator 
of the life-lived. It is the only lens with which we are 
able to know our concrete and abstract, real, and 
virtual world [1]. Here, | ask the reader to consider the 
wording, “affinity to the human body,” as “...only those 
elements that have some kind of affinity to the human 
body can [become known]. [1]" In order to build an 
experience for an actor, the designer must begin with 

a goal of something that can actually be experienced by 
the human actor. We have no affinity for cell division, 
or nuclear fission, or the digital change from a 1 to a 

0. That is to say, we have no way to feel these things, 
so if the goal is to make them known, to make them 
aesthetic, the designer has to first recognize this 
threshold and then offer a bodied interaction [2] to 


foster the knowing. 


Hyper focus on expanding rapidly evolving technologies 
without an overt attention to the human participation in 
these new landscapes runs the risk of debasing the very 
bodies that the technologies were invented to aid [3]. 

If we are to understand the semantics of technologies, 
then it must be through an understanding of human 
body-able form and movement, a Soma Literacy, as 

this is the only category of technologies that afford an 


experience [4]. 


2.1 Fundamentals of Experience 


In service of the above-mentioned attentions, 

the Carnegie Mellon School of Design required a 

new 8-week unit of study titled Fundamentals of 
Experience to all undergraduate Junior design students 
(Communication Design, Product Design, and 
Environments foci) in spring 2017 and 2018. The course 
introduced the design students to base concepts of 

felt interactions [4] recognized at the soma-tier of 
experience [5]. The methods were entirely interactive 
with no whiteboards, post-its, PowerPoints, or lectures 
utilized. The primary methodological disposition was 

to literally engage the students in the experiential 
concepts. The expectation was that as the designers 
developed a deeper soma literacy, they would be able to 
use this enhanced perspective to recognize, analyze, and 


manipulate (design with) the given soma-tier variables. 


In preparation for the course, we researched 
fundamental concepts of sentient experience such as 
tempo, cadence, beat, meter, range, crusis, phrase, 
rhythm, agogics, tension and release, rhythm, flow 
and interruption, ease and efficiency. We then looked 
for ways to engage the students with the concepts 

as participatory, extra-linguistic, kinesthetic and 


enkinesthetic [6] (social) experiences. 


The exercises were all based on the methods of 

Émile Jaques-Dalcroze [7], an early 20th century 

music pedagogue who used bodies in motion to build 
awareness to the felt experience of music. Throughout 
the eight sessions we utilized Dalcrozian methods to 
drive the student's attention to the felt experience 

of unfolding interactions. The strategies all involve 

the full group of students in simple, shared motions: 
walking, clapping, swaying, leaning, etc. The motions 
are regularly entrained to some outside force: a beat, 

a neighbor, the instructor, improvised music from a 
piano, prose, etc. Once some amount of synchronicity 
between the body of the participant and the outside 
indicator is achieved, the instructor then provokes the 
experience by manipulating any of the concepts being 
studied. By adjusting variables such as tempo, cadence, 
beat, meter, range, crusis, phrase, rhythm, agogics, etc., 
the participant is forced to either adjust their attention 
to maintain the entrainment or suffer the breakdown of 
the interaction. These methods fostered a redirection 
of the student's attention to the soma tier of their own 
experience as distinct from the aural, visual, or logical 


realities. 


2.2 An Introductory Lesson 


The first day of Fundamentals of Experience, the class 

was directed to simply walk about the room. Once we 
got over the micro-awkwardness of what seemed to 
many as an odd request, the instructor adjusted the 
exercise by building on the simple instruction and slowly 


morphing it into a deeper experience. 


The simple "walk about the room" was incrementally 
adjusted to include: "Walk with direction (Stake out your 
own path, do not walk in a circle)” . . . "Look where you are 
going (not at the floor)”. . . "See the peers that you pass" . 

. . “Greet a peer with a smile, a high-5, or a handshake as 
you pass”... "Adjust your tempo to match the tempo of the 
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music from the piano (teacher improvises music from the 


piano)" 


Over a period of about four minutes, we were able to 
progress from a directionless walking around the room 
to an enkinaesthetic series of interactions that required 
a specific kind of attention. With each small instruction, 
the students were nudged toward a more entrained, 
more embodied interaction and persuaded away from 
common biases of sight/sound/thought. The enactive 
techniques [8] fostered an attention to the bodied, the 


participatory, moving, shared experience with others. 


2.3 An Attention to the Interstitial 


On a separate occasion, the students were asked to 
listen to music from the piano...then to tap the beat... 
then to walk that beat...then to tap while walking the 
same beat...then to find a partner, face the partner and 
"patty-cake" the beat...then to press their hands on 
their partner's hands and push and pull, back and forth, 


in a sawing-type motion. 


The beginning instructions of "tap the beat" etc., 
revealed a common bias to the snapshot version of 
experience. When asking the class to demonstrate what 
they were paying attention to, the students first defined 
the beat by the moment of the hit, the moment when 
their hands came together, the crusis, the touchpoint. 
Through the expanding of the lesson to include the 
sawing motion shared with partners, we succeeded in 
revealing the felt space between these touchpoints and 
as a result, offered the students some insight into a 
ubiquitous phenomenon common to all felt interactions. 
We carried these insights (among many others!) into the 


analysis of experience design tools. 


2.4 Experience Analysis Tools 


Throughout the course we critiqued analytical 
tools commonly used in design to describe or plan 
experiences (i.e. journey maps and service blueprints 


[9 - 10] As the class acquired soma literacy skills, some 


1 For more examples of the exercises and projects completed 
please see pages 211-236, https://www.academia.edu/39225809/ 
SOMA LITERATE DESIGN. Recentering the Interstitiality - 


of Experience 
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of the shortcomings of these tools became apparent. 


While such tools are perfectly useful for charting out 
the successions of moments, they fall short if needed 
to demonstrate an attention to the interstitials chat are 


the hallmarks of the act of experiencing. 


Rather than focusing on snapshot touchpoint to 
touchpoint accounts of interactions, Soma Literacy 
values the felt, lived space between these touchpoints 
and as a result affords a view that speaks to the truly 


tangible, visceral, participatory elements of interactions 


The course concluded with the students proposing 
various revisions to such tools, attempting to include 
the interstitial trajectories revealed through their 
enhanced soma literacy. We then tested their revisions 
by proposing redesigns to common interactions (i.e. 
interactions such as riding the bus, progressing through 
a queue, following a recipe), then debating the role of 


the felt interstitial trajectories in significant experience. 


2.5 Lessons Learned 


With inspiration from the somaesthetics of Shusterman 
[11], performance as practice of Schwiebert [12], 
pedagogy of Jaques-Dalcroze [13], pragmatic philosophy 
of Wittgenstein [14], James [11], and Dewey [15], we 
presented an investigation of the body (soma) as the 


experiential core of perception and action [16]. 


The exercises permitted a depth of understanding 

that was not possible without engaging the student's 
actual sentient vessel, their own bodies in the knowing. 
Whereas traditional sketching permits the designer 

to explore two-dimensional space, and prototyping 
permits a sketching (or testing, playing, manipulating) 
in three-dimensional space, here we were interested in 


the analogous sketching in the fourth-dimensional space, 


Fig. 1. Leading design 
students in the sawing 
motion, Fundamentals of 


Experience 2018. 


Fig. 2. Carnegie 
Mellon Junior design 
students participating 
in the Fundamentals of 


Experience course 


time. And in these examples, where the attention was 
on significant experience, it is not merely time, but time 


as it implicates participating bodies. 


This study explored Soma Literacy as a tool of analysis, 
to notice how experiences, by default, design bodies. 
Seeing experience through this lens offers a different 
kind of body-ing from the majority of body-implicated 
design methods that use the body as a tool for ideation. 
While there are a number of design practices that use 
awareness of the body either as inspiration for designed 
interventions [17] (i.e. bodystorming [18], Wizard of Oz 
prototyping [19 - 20]) or as a model for other types of 
ideation (i.e. biomimicry [21]) the attentions that will 
permit the actor/designer an authentic experience in 
technology must be primarily concerned with the actual, 


in the moment body-in-motion. 


The initial runs of the Fundamentals of Experience course 
demonstrated that (1) there is an extra-linguistic tier 
of experience that students can recognize, (2) this 

tier of experience is full of distinct variables that can 

be isolated, analyzed, and manipulated, and (3) with 
even a basic understanding, the designer can offer 
interventions, nudging an actor(s) toward preferred 


states. 


Rather than only focusing on a logical dialectic knowing, 


these strategies supported kinaesthetic knowing by: 


* Presenting all of the content through group settings 


which not only encourage joint participation, but 


more-so a deep empathetic entraining of self-to-other, 
* Pairing students with multiple partners and small 
groups every class, 
* Using a spirit of play to push everyone to improvise, 
create, perform, share, and reflect in speed-round 


succession. 


3. Transdisciplinary in Design 


The human body-able technologies of form and 
movement, which include concepts such as flow, 
cadence, crusis, and agogics are central to any sort of 
participatory happening whether it be in manufacturing, 
web interfaces, theatrical performance, or ditch digging. 
Al and the surrounding technologies have no promise of 


aesthetic except to achieve these types of engagements 


[4]. 


*...designers should have knowledge of how to shape 
aesthetic interactions in a more visible, explicit, and 
designerly way. This is a kind of knowledge we are currently 
missing in [the interactive design fields] [22].” 


How do we adjust to an ever-evolving landscape of 


design attention? 


A first attention must be to the temporal. The 
deliverables of third and fourth order design are all 
expressed as happenings, experiences, and cycles [23]. 


Rather than things merely seen or touched, Design 
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for Service [24], Social Innovation [24], and Transition 
Design [25] all require solutions experienced as 
unfoldings and proceed with a tempo, a gait, a cadence, 
and a trajectory. This reality, requiring a valuing and 
attention to experience as unfolding time, opens a 
complex space and a need for a wholly different set of 


variables from the design rhetoric of the past. 


A second attention must be to more interest in, and 
technology for, the feeling body, as this body is 

not only implicated in, but central to the concept of 
interactions, experiences, and engagement. While the 
body in design is ever-more prevalent (as in motion 
sensing tech, AR/VR, or the "Soma-Based Theory" 
collective at CHI), it will only be through an explicit 
recognition of the body as a sensuous body, a body 

that experiences only through motion, where the 
connections will achieve aesthetic significance. If the 
practices are to discuss and design with efficiency, 
impact, and authenticity, an understanding, "sensory and 
cerebral, [...] characterized by an inward responsiveness 
to an outward stimulation [26]," becomes critical lest 
they separate the sentient actor even farther from their 


world. 


Here, at the crossroads of these two concepts: 
experience as unfolding time * sentient body, is where 
the Soma Literacy agenda becomes specifically relevant 
to design. Without an attention to and understanding 
of such concepts, the HCI and design communities will 
continue to working under an incomplete understanding 
of the role of aesthetics, and will be unable to 

efficiently discuss, analyze, and design with the soma 
tier of experience, inherent in every interaction and 

the proving ground for all significant participatory 


experience. 


4. A Soma Literacy for Designers 


“Thinking more broadly about this issue [...] it seems that 
we collectively have a very limited capacity to talk about and 
communicate haptic sensations clearly [27]." 


“We believe improved somatic empathy (through heightened 
body consciousness) could improve our ideation not only 
in movement-based interaction but in any interaction that 


deeply engages our body [28].” 
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The usage of the term soma comes from Richard 
Shusterman's sentient perceiving body-mind [29], 
a unified whole that is the active participant in 
an unfolding reality. Soma Literacy starts with 
acknowledging the body as the soma, noticing the 


feeling, dynamic, living body. 


Soma Literacy also notes that we come to know our 
worlds through a variety of registers of attention or 
tiers of experience (attentions to the visual, aural, 
logical, and somatic). Rather than a Cartesian mind/body 
split, Soma Literacy recognizes the embodiment of the 
actor, where one's thoughts, actions, and attentions 
are all implicated, one affecting the other, and even 
determining the other. The challenge of such attentions 
is in juggling one beside the other. Ontologically, we 
are embodied beings. Practically however, we develop 
habits and coping mechanisms that often skew and limit 
the noticing of the experiencing body. Common biases 
toward seeing, hearing, or thinking can run so rampant 
as to override the root discerner of experience, the 


resounding feeling in the sentient body. 


Lim, Stolterman, Jung, & Donaldson, wrote a short 
paper in 2007 titled, Interaction gestalt and the design of 
aesthetic interactions. In this paper they lay out a basic 


case for Soma Literacy in Design. 


*...the challenge here is to create a language that helps a 
designer understand which attributes are to be considered 
in order to create a certain gestalt that in turn will result in 
desired user experiences. [...] This language includes: (1) a 
good sense of what it is that is designed [...] in our case the 
interaction itself which we call interaction gestalt, (2) a good 
sense of what is possible for a designer to manipulate when 
designing the design target—in our case, the attributes 

of the interaction gestalt, and (3) a good sense of how to 
manipulate these attributes in order to shape a specific 
design [22]." 


Soma Literacy is just this, the understanding of the 
bodied implications of an interaction. Currently the 
fields of IXD have a very shallow discourse concerning 
interstitial spaces, performative entrainment, and 
awareness of the experiencing body. There is no 
agreed-upon understanding of the terms experience or 


embodiment as would be relevant to design, and there 


is no common listing of terms, concepts, or recognized 
palette of temporal variables. Literacy reveals what is 
hidden to the illiterate. Without an overt conversation 
of Soma Literacy, the design fields (and now | include 
music, dance, drama, cinema, painting, architecture, HCI 


and IxD) will struggle to comment on what is felt. 


Soma literacy is both (1) a way to analyze personal 
experience (analyze why a moment was or was 

not impactful) and (2) a way to create or foster an 
experience by willfully participating (or not) in the 
heavyllight that surrounds us or by designing an 
experience with these soma variables in mind. Soma 
Literacy gives one a specific lens to see through and 
participate in the world. It reveals tiers of experience 
that are hidden to many. This ongoing research attempts 
to highlight both the validity and the opportunities for 


the design fields in gaining such an understanding. 


5. Conclusion 


Deep technologies are now commonplace and raging 
forward. Separate from the engineering challenges of 
these initiatives are the challenges such technologies 
pose to our culture, society, and relations. If we aspire 
to experience complexity, or design for "connected and 
expressive artificial ecosystems [30],", if we are to 
design "with a particular focus on the aesthetics and 
human experience of these new systems [30]," then a 
reorienting of experiential attentions are necessary, a 


reorientation to the intimate, sentient living body. 


We are analog. All experience-ing is analog. Any 
implication of a digital experience is a misdirection 

which can only lead to a distancing of the human from 
their world. It is only through body-ing that one can 
come to know a technological world, as “...only those 
elements that have some kind of affinity to the human 
body can [become known] [1].” Our challenge is first to 
understand experience as the participatory kinesthetic 
and enkinesthetic action that it is, that is, develop a 
Soma Literacy, and then recognize the affordances 

of the burgeoning technologies that permit such an 
involvement. We must be continually reminded that the 
body is not capable of static being or of binary on/off 
positioning. The body is living. The body is analog. The 
body experiences over unfolding time and cannot jump 


from instant to instant (touchpoint to touchpoint, 1s 


to Os) separate from the interstitial swing-toward and 


swing-away-from. 


How do we adjust to an ever-evolving landscape of 
design attention? What new palettes do these modern 
fields obscure, reveal, and require? In our pilot course, 
Fundamentals of Experience, we explored base concepts 
of experience through personal engagement. The initial 
participatory lessons in Soma Literacy used methods 
revealing concepts outside of the traditional design 
education, yet potentially critical for the modern 
designer who is working evermore in explicitly temporal 
and technological realms. Regardless of the technology, 
if one is to consider the temporally unfolding aesthetics 
of any interaction, it will be only those artifacts that 
possess an affinity to the human body that can actually 


engage the human actor. 
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with and without Disabilities 


Abstract 


Technology integration in education has a great 
potential to transform learning paths, to overcome 
barriers and increase meaningful interactions among 
students, teachers, and the environment. Phygital 
learning is an emerging approach that balances the 
innovative technology-driven experience content 

with the traditional and physical one. In the context 

of Inclusive Education, this approach is particularly 
promising for enhancing the learning domains (cognitive, 
affective, and psychomotor) of children with disabilities. 
Recognized as integrators in multidisciplinary teams, 
characterized by a broad vision on users' needs and 
experience, and familiar with the creative problem- 
solving process, designers can have an active role in 
developing new learning activities. This paper describes 
the development of Magika, an interactive Multisensory 
Environment, that supports inclusive education via 
playful phygital (physical + digital) activities for children 
with and without disabilities. 30 specialists, among 
product and interior designers, electronic, materials 
and mechanical engineers, primary educators, therapist, 
and caregivers, were involved in a co-design process 

to define the educational and therapeutic objectives 

of phygital activities, according to the Italian primary 


school education system. 
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1. Introduction 


Inclusive Education is a fundamental point in the 

Italian education system, even though schools are 

not well equipped with spaces for learning, play, and 
socialization specifically designed for children with 
disabilities. In this context, our research aims at 
supporting and developing activities for children with 
physical and intellectual disabilities (ID). Ve design 
activities for LUDOMI project, one of the winners of 
the Politecnico Social Award of 2017 [36], whose goal is 
to provide multisensory smart rooms for schools. The 
project strengths are on the one hand the innovative 
interactions for users and on the other hand the 
multidisciplinary approach of the project, which allow 
merging different competences starting from system 
integration skills to expressive-sensorial properties 
knowledge of the materials. The output of the 
cooperative collaboration between therapist, educators, 
and caregivers is Magika, a smart multisensory room 
that supports inclusive education via different phygital 


(physical + digital) activities. 


Role of designer. By their nature, designers are 
connectors and facilitators in transdisciplinary dialogues 
between different knowledge [1]. The designer profile 
continues to evolve according with the complexity 

of the context in which he works, our society [2]: 
design discipline has become "adult" and its problem- 
solving process (design-thinking) is commonly applied 

by other professionals to design products, services 

and experiences, innovation processes, business 
models and strategies [3]. The evolution that industrial 
designers have experienced includes a significant change 
in practice, that is not limited to just design thinking 
[4]: from a strongly "artistic-intuitive" practice to 
research and validation-based one [5]. The involvement 
of designers in the processes of functional materials 
development [6—8], or in social innovation projects, 

is predominant also in companies which are examples 
of this change. Designers have always had a special 
relationship with materials and artifacts' materiality 
[9], [10]. Today, industrial designers play an active 

role in shifting towards a "radical" process of material 
development [11]: from practical experimentation 

to materials properties, from reflecting on materials 


quality to expanding the meaning of materials. 


Paper structure. This paper presents the conceptual 
framework used to design phygital activities that 
integrate the tactile sensory perception to empower 
learning through sensory-motor experiences. In this 
perspective, we want to shed light on the role of the 
designer in a technological environment for social 
innovation. Firstly, we describe the integration role of 
the designer in an interactive environment, then we 
focus on the main features of Magika, and we delineate 
the guidelines for designing phygital activities in a such 


innovative interactive environment. 


2. Smart Multisensory Room 


Background. Multisensory environment's rationale is 
grounded on the theories of embodied cognition and 
sensory integration that emphasizes the formative role 
of embodiment in the development of cognitive skills 
such as mental imagery, reasoning and problem-solving 
[33]. Specific interventions for persons with special 
needs aim at stimulating basic perceptual mechanisms 
and promoting perceptual learning [33]. They often 


take place in a dedicated Multi-Sensory Environment 


(often referred to as Snoezelen [34]) - a room equipped 
to provide gentle multisensory stimulations through 
sounds, lights, projections, soft fabrics, and materials. 
Prior research indicates that combining the physical 

and the digital world and offering multisensory stimuli 
through embodied interaction provides support for 
persons with disabilities. MEDIATE [35] generates 
sound and visual stimuli in response to gestures and 
footsteps on the floor in front of a large display and 
would stimulate low-functioning non-verbal children 
with autism. In this framework, in a multisensory room, 
we designed phygital (physical + digital) activities, 
which refer to tasks in a physical environment that a 
user can physically interact with to manipulate digital 
information. This work sheds light on the potential of 
phygital approach in a multisensory room. According 
to Antle [24] and Falcao [25], the possibilities provided 
by tangible interfaces, such as physical manipulation, 
physical-digital mappings, exploration, and collaboration, 
represent promising opportunities for learning. 
Eisenberg et al. [26] also support the fact that tangible 
technologies provide richer sensory experiences 
through the interweaving of computation and physical 
materials, extending the intellectual and emotional 
potential of people's artifacts and integrating expressive 
aspects of traditional educational technologies with 


educational properties of physical objects. 


Materials are at the core of any physical artifact, and 
not only they contribute to its function, but they also 
have aesthetic and emotional values that allow designers 
to shape the character of a product. Among different 
material features, expressive-sensorial properties [27] 
are the ones embodied in the 'skin' of a product [28] 
and can be related to users' experiences with and 
through materials [29]. These properties are usually, but 
not always consciously, evaluated by product designers 
when selecting materials and finishes. The surface 
features that can be perceived by the human senses as 
linked to a material's physical properties [30] are named 
sensorial properties. On the other hand, if such features 
are largely linked to a product's value and identity, or 
user experience and preference, they are identified as 
intangible properties. Karana et al. [31] focus on materials 
having a dual meaning, one that emphasizes the role of 
materials as being technical and the other experiential. 
One of the pioneers of the material experience both 


in a sensory and intangible way is Bruno Munari [32], 
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a designer and a design educator who gave great 


contribution to Design in reinforcing creative thinking in 


design education as experimental investigations. 


Introducing Magika. Magika [13] combines and 
extends the features of existing multisensory digital 
systems, proposing an inter-connected space where all 
children are involved in new forms of full-body, playful, 
multi-sensory, learning experiences. Magika's primary 
stakeholders are children from 4 to 10 years old, 
especially those with disabilities. They present physical, 
psychological, sensory impairments that cause a learning 


difficulty and a situation of marginalization [23]. 


Secondary stakeholders are support teachers, teachers, 
psychologists, therapists, eventually parents, and the 
community overall. We developed a technology, 
Magika, that transforms an empty room into a 
multisensory room (Figure 1) that integrates visual 
contents projected on the walls and the floor, ambient 
sound, smart physical objects, connected appliances, 
smart lights. These elements, controlled using a tablet 
and automated by a PC, react to children's manipulation 
and body movements (touchless interaction provided by 
a Microsoft Kinect) to offer visual, auditory, tactile, and 


olfactory stimuli in any sequence and combination. 


3. Guidelines for Phygital Activities Design 


Since Magika is a multisensory room where children 
explore the environment through different senses, 

we introduced tactile exploration. Phygital activities 
have been chosen as the proper channel to explore 
the physical world interacting with digital inputs and 
feedback, and, moreover, this environment gives the 


children the opportunity to learn sensory association 
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Fig. 1. The picture 
captures Magika 
environment with 
all its equipment, 
including material 
frames, smart 
objects, and RFID 
reader (ERA). 


connected to the tactile experience. In the next 
sections, we describe the design process of activities, 
starting from the objectives up to the definition of a 


phygital activity framework. 


Requirements Elicitation. We decided to test 

the phygital approach with therapists and teachers to 
clearly evaluate its potential. We decided to opt for a 
focus group to commit therapists and teachers directly 
and promote cooperation. Due to the long-term fruitful 
collaboration, all participants (3 therapists, 3 teachers, 
2 designers, 2 software engineers) felt comfortable 

and confident in speaking openly and frankly. The focus 
group started with a description phase, in which the 
multisensory room was shown. We asked specialists 

to impersonate into their therapy or educational 
session with children with and without disabilities, to 
exploit the potential of the room creating activities 
with materials freely without any forced interaction 
flow. Specialists tried to explore activities with available 
materials for 15 minutes each and took notes when 
required. We then met them in the focus group, and the 
moderator asked questions about their experience. The 
outcomes of this focus group allow us to design phygital 


activities in the multisensory room properly. 


Target. Activities could be played by both neurotypical 
and children with disabilities, among which not only 

the ones with physical, visual, or auditory impairments 
but also with intellectual disabilities (ID). ID is a term 
used to indicate a set of chronic deficits of cognitive, 
communicative, and social skills that limit learning, 
adaptive behavior in everyday life. ID is associated with 
neurodevelopmental disorders such as autism, Down 
Syndrome, or learning disorders. It involves 376 of the 


population in developed countries [18]. As regards 


Table. 1. Selected 
materials with their 


ranked properties 


children with visual and auditory impairments, they 
refer on the one hand to blindness, low vision and 
color-blindness and on the other hand to deafness and 
hard-of-hearing deficits. Moreover, we include children 
with inability to use a mouse or slow response time, and 
limited fine motor control. Our approach is focusing on 
offering alternative input devices to let the multisensory 


room accessible to everyone. 


Objective Taxonomy. Thanks to the focus group, 
we were able to understand and collect some basic 
educational objectives, from which we design phygital 


activities. We collected them in the following taxonomy: 


1. Primary school cognitive objectives: A. Basic logical 
categories; B. Space-time concepts; C. Measurement 
and size concept; D. Hypothesis and causality; E. Socio- 
affective capacity 

2. Disciplinary objectives: A. Chemistry: experiment and 
understand the properties of materials in a multi- 
sensorial context; B. Technology: learn to interact with 
digital and non-digital tools; 


3. Transversal objectives: A. Live multisensorial 


experiences; B. Participate in communication exchanges, 
learning the value of confrontation; C. Develop reflexive 
skills, adding personal relevance; D. Knowledge in 
action: learn to recognize sensory inputs and associate 


them with physical properties of materials. 


As a second step, a collaborative environment is 
required to let all the children with and without 
disabilities, both cognitive and physical, play together. 
For this reason, another important objective is to 


realize design-for-all activities. 


Material Selection. The material samples used 

in Magika were selected according to technical/ 
functional requirements and sensorial properties, 

based on a previous research project [14], where the 
relation between sensorial properties of materials and 
associated descriptors was investigated. First, selected 
materials shall be resistant enough not to be easily 
damaged by children when they are called to manipulate 
and explore them actively. Then, materials must fulfill 
hygienic standards, being washable and non-toxic. 


Another important aspect is that children cannot be 


Tag Colour Lightness Texture Softness Slipperiness | Deformability Transparency 
Al Pink 1 2 4 2 2 3 
A2 Orange 1 2 1 3 2 3 
A3 Light brown 1 2 2 3 3 3 
A4 Red 2 2 3 3 3 3 
A5 Light green 2 2 2 2 3 3 
A6 White 3 2 4 2 5 3 
B1 Light pink 2 1 5 2 2 
B2 Transparency 2 1 5 2 2 
B3 Blue 3 / 5 4 4 3 
B4 Black 3 3 5 4 3 3 
B5 Light grey 4 2 5 4 4 3 
B6 Light grey 4 1 5 4 3 
B7 Black 4 2 5 4 4 3 
C1 Green 2 / 3 5 3 3 
C2: Gold 1 5 / 3 
D1 White 3 3 3 2 2 
D2 White 3 5 2 2 2 
D3 Grey 2 4 1 1 3 
D4 Light grey 4 / 4 4 3 3 
D5 Blue 5 / 5 4 4 3 
D6 Black 5 / 3 5 3 3 
FA White 1 2 5 3 1 2 
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exposed to risk environment where they could get hurt, 
thus material selection is affected by non-breakable and 
non-sharp properties. At last, keep in mind our target 
is fundamental: experience for children with physical 

or visual impairments could be facilitated choosing 
material not too heavy or too bulky; as regards users 
with cognitive disabilities we expect some frustrated 
behaviours, thus material should be soft enough to 
avoid any risky situation. Following the above steps, 

30 material samples were selected, and then they 

have been classified by 5 material experts rating them 


according to 6 material properties (Table 1). 


The considered material properties are: lightness (Light- 
Heavy: 1-5 scale); texture (Rough-Smooth: 1-5 scale); 
softness (Soft-Hard: 1-5 scale); slipperi-ness (Slippery— 
Unslippery/No fluent: 1-5 scale); transparency 
(Transparent-Opaque: 1-3 scale); deformability (Stiff 

— Flexible: 1-5 scale). After ranking materials, they 

have been tagged to be recognized by the system with 
RFID tags, according to their material families (A — 
Polyurethane; B — Rubber; C — Composites; D — Fabric; 
E - Natural Material; F — Smart Material). 


Phygital Activity Framework. Since the main goal 
is to let children play and collaborate through the game 
and learning about basic knowledge of materials, we 
designed a framework for phygital activity: on-boarding 
phase, the first-time children enter the room, they 

are called to answer simple questions to profile the 
user; instructions activities, which aims at familiarizing 
with room and materials; tactile activities, which goal 

is to explore and learn materials properties. Three 
interaction are possible: the one with hands, the other 
with feet and the activities which include both. To 


provide a modular and flexible activity with materials, 
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cardboard-frames are produced: bigger frames for feet 
and smaller ones for hand. For the same reason, two 
dimensions of material samples, that can be inserted in 


the frames, are provided (Figure 2). 


The flexible approach encourages teachers to build and 
select their own materials set. To provide an engaging 
and likable user experience, the activities are explained 
and presented by an animated cartoon-avatar which 
looks like the cardboard frame of materials. The on- 
boarding phase is experienced just once followed by 
the instruction activities, which could be exploited 

any times teachers require. As concerns the tactile 
activities, they could be played every time, but they 
required that both on-boarding and instruction phases 
have been experienced, otherwise children would be 
not able to accomplish the task required. The tactile 


activities are structured in steps: 


-Explanation phase: the avatar defines what the activity 
is focused on (i.e., recognize the correct properties, 
identify the material association); 

-Exploration phase: the avatar shows to the child how to 
explore the material to recognize and identify correctly 
the associated property; 

-Instruction phase: the avatar gives the instructions to 
the child explaining which materials take to play with 
and which task he shall accomplish; 

-Action phase: the child fulfills the avatar requirements; 
-Reward phase: they receive the proper reward if acts 


correctly. 


The materials recognition system is RFID-based: each 
material is tagged with a label and the system, through 


a tangible object in the room, could identify if the child 


took the correct material from the material set [17]. 


Fig. 2. Material 
frames: each material 
is tagged, and it 

is designed in a 
modular way. Feet- 
frames (left — 33 cm 
x 33 cm), hand-frame 
(right — 10 cm x 10 


cm). 


Table. 2. Designed 
activities and their 


main features 


4. Results 


Design Phygital Activities. The on-boarding phase 
has been defined accurately for profiling users. 

Magika gathers data regarding their preferences about 
materials, not only referring to their likability but 

also question them about sound-material or image- 
material associations to adapt each activity to the user 
preferences, and to avoid any unexpected behaviour 
from children with disabilities (i.e., aggressive reactions). 
This information could also be used during tactile 
activities. Instruction activities are designed independently 
from that just to let children get familiar with the 

room. As a result of the focus group session, the tactile 


activities are: 


-Memory-tactile game: children shall identify and couple 
two items of the same material exploring them with 
their hands, and as classical memory game required, 
they shall remind their correct positions. 

-Twin-tactile game: children are required to match 

the material they explore by feet and the one they 
experience through hands. 

-Scaling game: children explore materials, and they are 


called to rank them according to the proposed property 


in crescent or descent order. This game provides a 
different difficulty level, which is adapted to the user 
knowledge and expertise. 

-Storytelling game: children are immersed in a story, 
where they are required to accomplish a specific task 
to go through the narration, mainly based on sensorial 


associations. 


Phygital activities have been designed following the 
objectives guidelines elicited by specialists discussed in 
the previous section. In Table 2, the activities and their 


main features are collected. 


Design Principles. To define design principles, we 
specifically asked specialists to express opinions on the 
usability of phygital activities and personal will to use it 
in everyday school or center-routine. We asked them 

to focus on the design and the mandatory features that 
might be added to activities in the room according to 
their expertise. All specialists were willing to contribute 
and pointed out some design guidelines when developing 
phygital activities in the form of potential requirements. 
The design principles elicited for the creation of phygital 


activities are: 


Designed activities Objectives Target Tools Players Design Principles 
On-boarding 1A, 1C, 1D, Children with and Digital projections, Individual Adaptivity, Aesthetics, 
2B, 3A without disabilities Sounds Design-for-all 
Instruction Activities | 1A, 1B, 1C, 1D, | Children with and Digital projections, Individual/ Aesthetics, Design-for-all 
2B, 3A, 3B without disabilities | Sounds, Hand/Foot- Group 
frames 
Tactile Memory- A, 1B, 1C, Children with and Digital projections, Individual/ Adaptivity, Aesthetics, 
Activities tactile D, 2A, 2B, | without disabilities | Sounds, Hand/Foot- Group Design-for-all, 
3A, 3B frames Controllability, Flexibility 
Twin-tactile A, 1B, 1C, Children with and Digital projections, Individual/ Adaptivity, Aesthetics, 
D, 2A, 2B, | without disabilities | Sounds, Hand/Foot- Group Design-for-all, 
3A, 3B frames Controllability, Flexibility 
Scaling game| A, 1B, 1C, Children with and Digital projections, Individual/ Adaptivity, Aesthetics, 
D, 2A, 2B, | without disabilities | Sounds, Hand/Foot- Group Design-for-all, 
3A, 3B, 3C, frames Controllability, Flexibility 
3D 
Storytelling] 1A, 1B, 1C, Children with and Digital projections, Individual/ Adaptivity, Aesthetics, 
1D, 2A, 2B, | without disabilities | Sounds, Hand/Foot- Group Design-for-all, 
3A, 3B, 3C, frames, Smell Controllability, Flexibility 
3D 
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-Adaptivity. Before starting to play in the room, the 
system needs to know which stimuli users like most and 
the ones which could create frustration to them: Magika 
collect this information for adapt the phygital activities 


according to the user during the on-boarding phase. 


-Aesthetics. In our project, this aspect is linked to the 
exploration of the expressive-sensorial properties of 
materials, generally used by designers to emotionally 
engage the user. Through the activities of stories, 

we try to create an immersive environment in which 
to educate to the multisensory experience through 
stimuli and visual-tactile and olfactory associations. 
The intelligent environment favors the repetition of 
associations preferred by the user and tries not to 


propose those critical to the subject. 


-Controllability. We have also designed a tool to tackle 
complexity in this technology-enriched system to let 
teachers control the multisensory room. The tablet- 
based app allows teachers to manage the whole 
experience and it contains three different sections: 
Create, where teacher can build their custom activities; 
Play, where teacher can choose which activities to 
perform during a session in the multisensory room; 
Live, which is a control panel thank to which teacher 
can go ahead during an activity even if the system is not 
working properly (i.e., Kinect is not detecting precisely 


the child position). 


-Design-for-all. Activities are designed to provide 
information to children through different channels, both 
visual and auditive. Following the principle of design-for- 
all, Magika becomes an inclusive environment where 
other children adjust their interaction modes according 
to the ones with impairments (reverse-inclusion). 
Multisensory environments also allow a wide range of 
people with different perceptive and cognitive abilities 


to play actively in the room. 


-Flexibility. The teacher can introduce new custom 
materials which could be used during the activities in 
the room. They are free to enlarge the material kit and 
create new activity according to their needs using the 


Create section of Magika tablet-based app. 
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Designers are required to multidisciplinary and 
openness towards an evolving system design (in 

our case activities). In fact, we did not think about 
activities related only to the materials provided but 
based on the preferences of the children or those of 
the teacher, trying to encourage the implementation 
of new materials. The critical aspect is their use in 
property classification activities; however, they could 
be used in stories and/or other games (or relaxation, 
etc.). Since the two rooms have been installed in two 
primary schools, we tested our phygital activities in 

an educational environment. Children performed an 
explorative session where they played with the designed 
activities in the room. All design principles played a key 
role in supporting children activities in the room. A 
methodical experimental study will be performed for 
providing evidence on the effectiveness of the design 


process. 


5. Conclusion and Future Work 


To achieve its goals, the project must necessarily 
embrace a multidisciplinary approach. In technological 
terms, the project requires advanced system 
integration, Internet of Things, physical systems, 
interactive olfactory systems, and physical exploration 
of materials. To design the Magika physical space, 

the "smart" objects, and their interactive properties, 
different skills from the design field are needed (UX, 
UI, interior and product design) [16]. The definition 

of stimuli and activities for children involves the 
educational and therapeutic skills of professionals in 
disability. These multidisciplinary requirements guided 
the definition of the working group and the partners. 
The innovation of the project is also in experimentation, 
unique in its kind for complexity and size. The empirical 
validations of interactive technology for individuals with 
disabilities are intrinsically very difficult, and the total of 
participants in existing empirical studies varies between 
3 and 20. The project team has installed two Magika 
Room solutions in real contexts (two primary schools 
in suburbs-city) and will conduct an experiment on the 
field involving over 800 children with disabilities of 10 
municipalities in the Milan suburbs, 58 teachers, and 

51 classes. In this complex digital-physical ecosystem, 

a schematic experimental approach will be adopted to 


collect valuable information regarding user experience. 
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Recommendations when Designing to 
Address Procrastination: A Psychological 


Perspective 


Abstract 


Procrastination is a common behaviour that 
psychologists have found to have many negative 
consequences for both the individual and society. 
Standard psychological methods for addressing 
procrastination require significant time and effort, and 
consequently suffer a lack of adherence. This paper 
synthesises relevant psychological research to identify 
possible approaches designers could take in order to 
offer immediate aid to procrastinators. We suggest 
that an understanding of the psychological mechanisms 
underlying procrastination may inform and guide 
designers in creating interventions that shift some of 
the effort associated with undertaking tasks from the 
individual to the designed environment. In the paper, 
we draw on different psychological perspectives and 
strategies, highlighting how this information may be 
relevant and applicable for designers who aim to address 
and reduce procrastination behaviour through designed 


interventions. 


Keywords 


Procrastination, Design, Psychology, Motivation, 


Attention 


1. Introduction 


We all find ourselves putting off things that we would 
have benefited from doing earlier. For the many, 
procrastination is occasional —a few tasks without 
deadlines never get done and some tasks are done 

last minute — but generally the procrastination 

does not have a big impact. However for some 
populations like students, (85-95% of whom report 
procrastination[1]), and those with high autonomy 
over their time [2], procrastination is prevalent and 
can have a large impact; affecting mental health and job 
performance. For 20 percent of the adult population, 
procrastination is habitual [3] impacting all aspects of 
life. The current psychological treatments for reducing 
procrastination take practice and effort [4], which can 
be particularly difficult for procrastinators. Almost 
half of those starting cognitive behavioural therapy, 

a common treatment, drop out before progress is 
made [5]. Designers may be able to help people reduce 
procrastination in a more immediate way by creating 
environments that counteract underlying causes. In 

a cross-disciplinary endeavour, this paper presents 
designers with the relevant psychological theories to 
consider when designing interactions that navigate the 


complexity of procrastination. 


Putting tasks off does not necessarily constitute 


procrastination. There are many good reasons to delay 
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a task: sometimes it is beneficial to wait in order to 

work with the latest information; sometimes another 
task is more important; sometimes a person needs a 
break first [6]. As Steel defines it, procrastination as 


the needless delay of a task to one's future detriment 


[7]. 


Procrastination is correlated with many undesirable 
outcomes. Procrastinators tend to have shorter-term, 
lower-paying jobs and make up nearly 60 percent 

of the unemployed [8]. Procrastinators often have 
poorer general mental health, lower self-esteem, 
lower self-efficacy, higher levels of neuroticism [7] 
increased stress [9], more long-term unhappiness [2], 
higher levels of self-sabotage[7], and higher levels of 
guilt due to breaking social norms [10]. In spite of this 
many people argue procrastination helps them: “I work 


9, 66 


better under pressure"; “it helps focus me". However, 
empirical studies present a different story [9], [11], 
[12]. People make more mistakes, are less creative, 

and enjoy the overall experience less, both while they 
are delaying the task and when they are doing it under 
time pressure[13]. As Pychyl puts it, ‘People don't work 
"better" under pressure; it's just they "only" work 


under pressure'[6]. 


Procrastination is also costly to society. In addition 

to ‘putting work off’, people procrastinate in many 
contexts of life, including health and wellbeing [14], 

life transitions such as retiring from work [15] and 

their personal lives [6]. Putting off health care 

choices [14], retirement savings [15], and on average 
procrastinating twenty-five percent of the work day [2], 
means that employers and taxpayers have to make up 
the difference. Arguably, if designers can help people 
address their procrastination, then both the individual 


and society will benefit. 


Procrastination is often viewed as the result of laziness 
or lack of planning. However, as Steel’s review of 
procrastination studies shows, procrastination results 
from failures in the self-regulation of emotion, attention, 
motivation, or engagement [7]. Psychologists typically 
address self-regulatory failure through cognitive 
behavioural therapies, restructuring negative thoughts 
and behaviour patterns [16], [17] teaching specific 
goal-directed behaviours [18], [19], implementation 


intentions [20] and Applied Behavioural Analysis [21]. 
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Psychologists also employ cognitive strategies to 
increase self-efficacy, reduce negative thought cycles, 
self-handicapping, and irrational beliefs [19], [22]. These 
strategies all address the underlying regulatory failure 
but require significant time and effort to implement. 
Though 95 percent of procrastinators express a desire 
to change [9], many are discouraged and unable to 
adhere to therapeutic practice because the focus is on 
long-term change rather immediate aid[5]. This paper 
focuses on psychological approaches that designers 
may operationalise through the shaping of the physical 
environment, to offer people more immediate aid, 

as well as supporting the process of learning non- 
procrastination. We are particularly concentrating on 
work-based tasks conducted at a computer. There are 
other ways of affecting how a task is done through 
design that are outside the scope of our consideration, 
for example, through: changing the task itself; changing 
the context of doing the task; changing the user. We 
focus here on making change through the design of the 


built environment and the interactive artefacts within it. 


A review of the literature in conjunction with Blunt and 

Pychyl's work [23] highlights three types of task that 

people commonly procrastinate in the process of doing: 

* Tasks that cause anxiety: It might be that people are 
unsure of what is involved in the task, it might be that 
they are unsure of their ability to do it, or it might be 
the consequences of failing are high; but all of these 
cause anxiety that people want to avoid; 

* Tasks that are tedious: These tasks may not be hard 
but require people to focus on something which isn't 
engaging or stimulating; 

* Tasks that are effortful for low perceived reward: 
These tasks tend to be frustrating because, in spite 
of consuming a lot of effort, people get very little in 


return. 


The cognitive view of procrastination as a failure in self- 
regulation means, in plain terms, that a procrastinator 
has been unable to make themselves do an undesirable 
task rather than do a more enjoyable one [7], [19]. The 
‘self-regulation failure’ involved in procrastination can be 
described from a range of cognitive perspectives. The 
perspectives of willpower [24] and emotion regulation 
[10], [25] are highlighted in this paper in terms of their 


potential relevance for designers. 


2. Willpower Perspective 


Psychologists have found that willpower is a finite 
resource [24], [26]. As people go through the day, their 
willpower is slowly used up, regardless of how it is used. 
To forgo distractions, people use up willpower. When 
people have many pleasurable distractions surrounding 
them, their willpower resources can be quickly depleted 
[26]. Technology design now provides many instant 

‘fun distractions’ (e.g. smartphone game applications 

or social network sites), making it increasingly hard for 
procrastinators to attend to the task at hand. Their 
willpower is constantly being called upon and therefore 
is readily used up [6], [26], [27]. People also have 

many work-based distractions, with constant alerts 
making it easy to switch their attention away to simpler 
tasks [28], [29]. To address procrastination from the 
*willpower perspective', designers need to look at how 
to reduce the amount of willpower required for each 
type of task. Willpower is used for both starting and 


maintaining task engagement. 


The following will argue that designers can reduce the 
willpower requirements by: 

* Making the task more desirable; 

* Making the distractions less desirable; 

* Using pre-decisions (‘implementation intentions’) to 


reduce periods of high demand on willpower. 


2.1 Making Tasks Seem More Desirable 


Feasibly, designers can increase the desirability of work 
tasks using several strategies. One strategy is to design 
for Flow conditions that are associated with enjoyable 
and intrinsically rewarding experiences [30]. Flow is 
commonly used by interaction designers to increase 
engagement. One aspect of Flow is knowing how well 
you are progressing [31]. We suggest that designed 
environments could provide people with feedback on 
their progress offering them the opportunity to feel 
good about it. This could be particularly effective for 


tasks which have long-term rewards. 


Other key aspects of Flow are knowing ‘what’ and 
‘how’ to do the task at hand and having the skills to 
accomplish it [31]. Designed interactions could help 
people break large tasks into more achievable steps 
helping increase their confidence and use their current 


skill levels in order to experience Flow. However, if 


Flow is, fundamentally, about balancing challenge levels 
and individual skill levels[32], then it is important to 
acknowledge that sometimes people have to do tasks 
where this is not the case. Consequently, designers 
may need to consider another strategy. Rather than 
designing for Flow, they could increase task desirability 
by adapting the environment. Creating environments 
and interactions that users find desirable, may make 
tasks seem more enjoyable (or less disagreeable). 
Grading by a fire, with a hot drink steaming beside you 
seems more appealing than sitting in a hard chair, under 


fluorescent lights. 


Starting a task is the most difficult part, and distractions 
can cause people to 'start' (re-engage) several times, 
even within one working session [6], [13]. Once people 
have started, the distractions and fears that were 
motivating them to procrastinate are less noticeable and 
they often find the task more pleasant than expected 
[13]. By creating pleasurable interactions when initiating 
work designers may be able to use the salience of 
starting a task to increase task desirability. Experiments 
investigating cognitive abilities during dual tasks[33] 
indicate that interactions on different sensory channels 
(touch, site etc) to the main task can be processed 
without interfering with work[34]. We suggest that 
designers be careful not to create interactions that 
could present barriers to the work, and to consider 
how the experience could change over time, to avoid 
mundanity. We also predict that different solutions may 
be needed for 'starting a task' versus 'restarting after a 


distraction’. 


Alternatively, designers could increase desirability 
through adding extrinsic rewards —a core component 
of gamification. Gamification aims to replicate the 
principles of gaming in non-gaming contexts and is 
commonly used by designers to increase engagement 
[35]. Extrinsic motivation is known to be helpful for 
changing behaviour [36]—[38] and can be attractive 

as it seems simple to design a reward system. Still, 

we advise designers to look carefully at schedules 

of reinforcement and interval times [39]; just a few 
milliseconds difference in timing rewards can make 
the difference between a successful system and a 
gimmick that loses its appeal. Notably, several studies 
indicate that extrinsic rewards may reduce intrinsic 


motivation [38], [40]. These results are not conclusive 
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[41] but do highlight the complex relationships between 
different types of motivation. If using extrinsic rewards, 
we recommend considering social rewards that are 
concurrent to the tasks. Social rewards are hardwired 
for neurotypical people and are strong motivators 

[42]. We suggest offering social rewards during the 
task rather than the end, which could add to the task 
pressure consequently increase procrastination [10], 
[43]. We conjecture that concurrent rewards may mimic 
intrinsic motivation so people judge the task itself as 
enjoyable. This is why having a ‘study buddy’ can be so 
effective[44]. 


2.2 Make the Distractions Less Desirable 


Removing Distractions. Designers may be able to 
reduce the required willpower by removing distractions, 
thus making them less desirable in the moment— 
studies show that “Out of sight, out of mind" isn't just 
a saying [6], [27], [45]. Online Procrastination guides 
[6], [46], [47] suggest using tools like website blockers 
or devices like Saent [48]. We conjecture that some 
people find it easier to work in public spaces [44], 
because the social norms limit some of the distractions. 
Interestingly, the act of removing the distractions can 
be as difficult as starting the task [49], particularly if a 
person's leisure and work share the same space. We 
propose that making distractions easy to remove and 
not associated with 'depriving oneself from pleasure' 


may be a helpful approach. 


Drowning Out Distractions. On the whole 
external distractions can be removed or avoided. 
However, internal distractors (thoughts) cannot be 
simply removed. It takes willpower and effort not to 
follow a daydream or give in to self-doubt [6]. Stress 
narrows peoples' attentional focus [50—52] which 

can aid in ignoring internal distractors—an apparent 
benefit of working last minute. Unfortunately, this 
narrowed attentional focus can also result in poorer 
work outcomes and missed opportunities [51], 

[52]. Designers may be able to recreate the positive 
aspects of narrow attentional focus by using sensory 
stimulation to ‘drown out distractions’. According to 
the Perceptual Load Theory of attention processing, 
overloading perceptual inputs, may reduce vulnerability 
to distractions as there is no perceptual capacity left to 


process new information. Murphy, Groeger & Greene 
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[53] give an excellent example: if a person is reading, 

a buzzing fly is likely to pull their attention away from 
their book. However, if the text is printed on mostly 
transparent paper with words on both sides, their 
sensory attention has to work hard to decipher the 
letters and the person is not likely to notice the fly. As 
long as the additional sensory inputs do not require 
cognitive attention, they generally do not interfere with 
cognitive performance [34], [54]. We conjecture that 
designed environments or artefacts could use sensory 
channels to pleasantly overwhelm a person's perceptual 
processing. The sensory inputs will need to be subtle 
enough not to divert attention from the primary task. 
For example, some people find sounds or music helpful 
when writing, but find songs with lyrics distracting. 
Companies like Fuzeinteriors [55] already place high 
important on the sensory experience of work spaces. 
And installations like Sensorium by Les M Studio [56] 


offer excellent examples of engaging touch. 


Negative reinforcement. Alternatively, designers 
could use negative reinforcement as a motivation 
strategy [57—59]. Principles of negative reinforcement 
suggest a ‘less pleasant environment’ when people 

are ‘off task’, will make distractions less appealing 

and motivate people to focus. For example, when a 
person is ‘off task’, the chair they are sitting on could 
react becoming bumpy and less comfortable; and when 
attention is redirected to the task, the chair could 


become comfortable again. 


When reviewing technologies that have been designed 
to address procrastination, we found negative 
reinforcement was a popular strategy. The smartphone 
application (app) “Yelling Mom” is a prime example; 

an alarm yells at the user until they do their work 

[60]. We identify issues with this approach, negative 
reinforcement becomes ineffective if it can easily be 
avoided: if the user can turn off the phone more easily 
than starting an assignment, then it is not likely to 

add motivation. Again we invite designers to consider 
carefully how negative reinforcers may become a barrier 
to the task: interventions where it easier to leave the 
workspace than to start work may, in fact, reinforce 


procrastination. 


Scaffolding. It is possible, though not necessarily 


advisable, to use guilt as a negative reinforcer. Anderson 
suggests 'scaffolding' to force non-procrastination 

[45], [61]. He suggests creating environments that 
constantly remind people to do their work [61]. The 
presence of clocks, reminders and post-it notes makes 
it difficult to dismiss work or justify delay. Though 

quite possibly effective, we are wary of this approach, 
on the presumption that it is not necessarily conducive 
to a pleasurable work experience. Motivation from 
guilt, rather than enjoyment, will likely not boost 
positive emotional associations, which in turn would 
not increase intrinsic motivation. If designers were to 
consider using this motivation strategy, we suggest 
integrating periods of time free from pressure so people 
can experience positive downtime. Additionally, if the 
designed interventions ‘invite’ people to work, rather 
than 'pressure' them to, then the associated guilt may 


be reduced; but the reminders would still be there. 


2.3 Implementation Intentions (Pre-Decision 
Making) 


Finally, people can reduce willpower by spreading 

out the decision making. Implementation intentions 

is a psychological strategy employing pre-decisions 

to reduce spikes in cognitive load [62]. In the case of 
procrastination, the procrastinator would make the 
decision of when and what work to do beforehand. 
When the time comes to act, the pre-made decision 
requires less cognition. Implementation intentions are 
most effective when pre-decisions for distractors have 
also been made [63]. For example: "Even if | don't feel 
like starting, | won't open the news, not even for one 
article"; or “if Facebook is still open from yesterday, | 
will close it before looking at the timeline". These pre- 
decisions makes the cognitive process simpler when 
the event happens [64]. There is an opportunity for 
Designers to create physical reminders of pre-decisions. 
For example, a coffee table might automatically 
transform into a desk at the time the person intends to 
work. We expect that physical reminders will not only 
reinforce the pre-decisions but also make it harder for 
people to ignore them (similar to non-procrastination 


scaffolding). 


3 Emotion Regulation Perspective 


From an 'emotion regulation perspective', people tend 


to procrastinate to avoid the negative emotions they 


experience when approaching a task [25]. The three 
types of tasks evoke different emotions so they should 


be addressed in different ways. 


3.1 Effortful with Little Reward 


Tasks which are effortful with little reward evoke 
frustration. Such tasks could be helped by reframing 
them so their value and rewards are perceived better 
[50], [65]. How to reframe will vary. Herein, we 
recommend focusing on latent benefits of the task, or 
the accomplishment of doing the task. Anecdotally, 
people find that counting focused hours or progress is 
useful way of feeling good. We suggest designers could 
reward users with subtle changes in the environment as 
they work to highlight the passing of time or progress. 
(This also links with one of the aspects of Flow 
mentioned earlier [31]). When highlighting the passing 
of time, be careful it doesn't cause people to regret the 


hours they have put into unrewarding tasks. 


Simple extrinsic rewards could also be used to make 
the outcome appear more worthwhile [36]. Though as 
mentioned earlier, there's a danger of the task becoming 
even harder when the rewards are no longer present. 

If designers were to employ gamification we propose 
carefully developing the rewards that fit the specific task 
and context: rewards that appear condescending will 


make the situation worse. 


Conversely, a designer could address the frustration 
rather than the reward. Many people find grumbling 
and sharing their frustrations to be a release because it 
allows them to acknowledge their emotions [66], [67]. 
As long as this doesn't become rumination, wallowing, 
or a form of procrastinating, then we think a designed 
interaction for people to express their frustrations may 


help them move on to the task. 


3.2 Tedious Tasks 


Simple tasks that are tedious may be addressed 

by making the process itself more stimulating and 
enjoyable. The performance of tedious clerical tasks 
were improved by just adding the smell of peppermint 
to the environment [68]. Designers could engage a 
variety of sensory inputs, but should be wary of not 
slowing the process as this will increase the tedium. 


We also suspect that if people see the stimulation as 
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gimmicky, they are likely to reclass the task as effortful 


with little reward. 


The theory of optimal stimulation may be particularly 
useful for understanding people's behaviours during 
tedious tasks. Optimal stimulation states that people 
function best at a specific stimulation level [69], [70]. If 
people's stimulation is below this, they self-stimulate, 
similar to those with ADHD, seeking out new engaging 
activities or engaging their senses through movement 
and sound. If individuals are over stimulated then 

they withdraw, similar to those with autism, focusing 
on small details and creating ‘white noise’ sensations 
to drown out inputs. We expect that if designers can 
increase peoples' stimulation during tedious tasks, then 
their perceived boredom and their desire to redirect 


their attention will decrease. 


3.3 Anxiety 


People find tasks that cause anxiety particularly difficult. 
Generally, the best way to reduce the anxiety is for the 
person to start the task and discover that they are in 
fact capable. Interestingly, experience sampling studies 
have found that people feel happier when they are doing 
the task they have been putting off [13], [71]. In fact, 
people often wish they had started earlier so they "had 
more time to spend on the interesting aspects"[13]. 
The anxiety associated with task engagement is 

often not acute anxiety but low-level anxiety caused 

by uncertainty and the unknown. We identify that 
designers have two general approaches open to them. 
Either they can reduce the anxiety directly or can they 
distract people from it, aiding progress before they give 
up. We speculate, working last minute can be attractive 


because people no longer have time to worry. 


To address anxiety directly, designers may use 
physiological approaches to activate the para- 
sympathetic nervous system — encouraging people 
to breath more slowly and assume relaxed body 
postures[72], [73]. Other than exercise, this is one 
of the most reliable ways to reduce stress without 
addressing internal thoughts [68]. Many applications 
already exist to support relaxed breathing[74]. 


Further, environments could be designed to counteract 


the anxiety, by evoking associations of comfort and 
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social support. Social support has been shown to 

be a significant moderator in stress [75]. A designer 
could create environments that remind people of their 
support networks during a task or they could provide 
communication channels to facilitate social support in 
work environments. There is also potential for design 
interventions to directly simulate social support. Touch 
is a vital aspect of human connection [76]. Social touch 
is as effective at communicating emotion as facial 
expressions and is stronger than verbal communication 
[76]. Social touch has been shown to increase 
compliance in some cases, and decrease anxiety, 

as well as raise serotonin levels and general health 
outcomes [76]. Simulating touch may allow people to 
experience the effects of social touch even without 
others. Several attempts have been made to simulate 
touch, particularly in autism research after Temple 
Grainden's hug machine[76]-[78]. These devices have 
not been empirically tested, although the techniques 
are widely used, and products such as weighted 
blankets are common in the autism community. There 
is an opportunity for designers to analyse the aspects 
of a comforting touch and then apply them through 
non-human mediums. Interactive furniture that subtly 
hugs the user may be help people feel supported and 
comforted. Designs like Hugvie®[79] and TapTap [80] 
are used in the context of remote communication, and 


when empirically tested indicate drops in cortisol[81]. 


Distraction from emotions is a common (though not 
always healthy) way to regulate emotions. Pychyl and 
Sirois described procrastination as 'maladaptive emotion 
regulation'[25]. By avoiding the task, the procrastinator 
avoids negative emotions. We think helping people 
sidestep anxiety may be the most effective way to 

offer immediate aid. Over time this will build positive 
associations and hopefully reduce the anxiety. For 
procrastination, ignoring emotions can be positive 
because doing the task addresses one cause of the 


anxiety. 


To distract from emotions, the theories of optimal 
stimulation [70] and perceptual load [53] suggest that 
people may be able to direct 'spare' attention towards 
low-level stimulation rather than their anxious thoughts. 
If designers were to combine emotion distraction 


with a social touch they may be able to create a low- 


level sensory stimulation that shifts attention, as well 

as giving an emotional boost. We would encourage 
designing two-way interactions that give both the user 
and object a prominent role. For example, A desk could 
react to a user's nervous fidgeting and by squeezing 
their hand simulating a social touch. Similar to TapTap 
[80]. 


5. Developing Research and Conclusion 


The previous sections have contributed to design 
research by outlining the ways that a design space for 
addressing procrastination can be approached and 
informed by psychological perspectives. The general 
strategies proposed herein could be tailored for specific 
contexts and tasks. Our current research takes these 
strategies and applies them to the context of tertiary 
students working at home. We are focusing particularly 
on students because: firstly, they have the highest levels 
of procrastination [7]; secondly, because they are still 
learning to self-regulate and, if we can help develop 
positive strategies early, they will hopefully carry on 

in later life [82]: and thirdly, they do not have fixed 
work spaces which means they often work in spaces 
that are also for used for leisure. Spaces used for 
leisure are particularly hard to work in because they 
present habituated distractions and do not trigger work 
behaviours through associations [83]. We aim to tailor 
the design interventions for the different stages of task 
engagement. Interventions to start a task will differ 
from those to maintain a task. We also acknowledge 
that finishing is a crucial part of the process. At the end 
of a work session, people can feel enjoyment, pleasure 
and pride, which are key to increasing their positive 
associations with work. This has design implications, for 
emphasising the feeling of achievement through designed 


interactions. 


This paper provides designers with a grounding in the 
some of the psychological theories of procrastination. 
It offers suggestions for how these insights can inform 
and guide the design of the physical environment and 
interactive artefacts within it, in order to address 
procrastination. Design strategies will differ depending 
on whether the designer addresses willpower or 
emotion regulation, and depending on what type of 
tasks they want to support. We hope that this cross 


disciplinary synthesis will provide a framework for 


design interventions that inspire and challenge tangible 


research outcomes. 
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R25: Designing a Public Augmented Printed 
Media System to Promote Care Home 
Residents Social Interaction 


Abstract 


Social interaction between care home residents was 
found to be limited and positive peer relationships 
were difficult to establish. Our research explores the 
potential of designing socio-technical interventions 
through digitally augmenting residents’ daily activities 
in public spaces. In this paper, we present ‘R2S’, a 
system aiming to support residents’ social interaction 
by augmenting public print media in care homes. We 
describe the design motivations, design process and a 
user trial of R2S. The result indicates a promising design 
direction to create shared experiences by augmenting 
individual behaviors in care homes. Implications are 
also discussed in three levels to inform future design of 


social technologies in public caring environments 
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Augmented Reality 


1. Introduction 


Positive peer relationships have been repeatedly 
identified as an essential element of care home 
residents’ quality of life [1]. However, in the past 


decades, in spite of improving physical caring 


environments, boredom, loneliness, and helplessness 
were still reported as common problems in care homes 
[2]. Although researchers suggested that residents 
should go to the public areas more often to engage with 
others [3], many surveys indicated that most residents 
still spent a large portion of their time in their private 
rooms alone and inactive [4], which is a strong predictor 
of many mental diseases [5]. Hosting various scheduled 
activities has long been recognized as the main solution 
to attract residents to go to public spaces, but this 

kind of intervention has many limitations [6]. Besides, 
Roberts et al. (2015) found that the social relationships 
between residents were mainly established through 
informal and unplanned activities [7]. Claessens (2013) 
also argued for the importance of providing social 
opportunities rather than enforced sociability [8]. 
Therefore, more innovative ways are needed to make 
the public caring environments more attractive and 
engaging to support residents’ social interaction [2]. In 
recent years, various public systems have been designed 
to socially connect people belonging to one community. 
However, older users have long been playing a minor 
role in designing novel technologies. Research and 
implementation of such technological interventions 

are scant, especially among institutionalized older 
adults. Our research aims to provide knowledge and 
implications of designing socio-technical applications 

in residential care settings. In the following part, we 


firstly learned lessons from previous work, and propose 
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a design direction to promote residents' shared 
experience by digitally augmenting their daily routines. 
Then, we present our exploration of designing a public 
augmented print media system (R2S). A user trial 

was conducted to collect initial feedback and further 
refine the system. The findings could inform future 
research and practice on designing technological social 


interventions in caring environments. 


2. Motivation from Previous Work 


The early explorations of socio-technical interventions 
in care settings could be traced back to the 1990s when 
researchers sought to reduce demented nursing home 
residents' agitated behaviors by adding visual, auditory, 
and olfactory stimuli to simulate different types of 
environments inside nursing homes [9]. With the 
proliferation of modern technologies and the increasing 
demands for high-quality care facilities, socio-technical 
interventions have been explored in diverse forms. One 
branch of this area attempted to develop assistive social 
robots that function not only as an interface for older 
people to digital information but also an intermediary 
agent to connect them by providing companionship 
[10]. However, many studies identified older adults' 
reluctance toward robots with a social purpose for 
being non-authentic in their expressions and interaction 
with humans [11]. Our research mainly followed another 
direction where the explorations aimed to influence 
residents' social behavior or feelings by designing shared 
displays [14]. For example, Gaver et al. (2011) designed 
a mobile device called "The Photostroller" to support 
residential social care by continuously displaying a 
slideshow of images from social media [12]. Residents 
can select image categories with a removable control 
unit. Kang et al. (2018) developed an interactive gallery 
called “OutLook” that can display real-time views 

of typical local places [13]. Both of the case studies 
conducted field trials and found a similar problem that 
many residents tended to passively watch instead of 
actively interacting with the displays. They rarely used 
the interactive features without the facilitation from 
caregivers, family members or designers. The main 
reason was concluded that the two systems failed to 
fully blend in residents' living environments and stable 
daily routines, which would decrease their time and 
frequency to interact with technologies and other 


people. Therefore, we assume that, since it is difficult 
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to change their long-term habits, why don't we enhance 
their current daily routines instead of introducing more 
new interfaces or interactions? Can "less is more" 

be a principle in designing social technologies in care 
settings? We believe it could be a promising direction to 


explore. 


According to many observational studies, reading is 
one of the most common behaviors in public caring 
environments [2, 4]. Various kinds of print media 
(books, magazines, newspapers, etc.) are usually 
provided to continuously support residents' daily 
entertainment in public areas. However, restricted by 
the conventional form, most print media products are 
designed for individuals. Reading and communication 
rarely take place simultaneously. Furthermore, due to 
the physical and mental degradations, reading is getting 
increasingly challenging and less attractive for many 
older people, which has the potential to be augmented 
by larger displays and dynamic forms of digital 


information. 


Given the opportunities and challenges, we turned to 
explore the design potential of augmenting print media 
by technologies to increase the attractiveness of public 
print media and promote shared experiences. Although 
the technological solutions to augment print media have 
been extensively explored in the fields of education, 
exhibition, commercial promotion and entertainment 
[16, 17, 18, 19], most of them had certain requirements 
on users' precision and proficiency, which was very 
difficult to be accepted and easily used by older users 
[15]. Moreover, most of these studies lacked further 
considerations of social contexts, especially in caring 


environments. 


3. Design Process 


The design process consists of a preliminary study 

and participatory design sessions. Both of them were 
conducted in the open areas of Dutch care homes 
belonging to a caring organization in Eindhoven. It has 
set up 22 nursing homes with similar environments and 
services in this city. Although most participants had 
physical disabilities or sensory impairments in varying 
degrees, all of them had normal cognitive functions 
because the care homes had separate areas for the 


residents with dementia. 


3.1 Preliminary Study 


Before designing the system, several questions need 
to be addressed. For example, among so many kinds 
of print media provided in nursing homes, which 
have the potential to be augmented? Which genres 
of print media products have the potential to trigger 
residents' social interaction? To answer them, we 
started with a contextual study to investigate 21 
residents' use of print media, preferred genres and 
their related social demands and barriers [6]. The 
method used was a combination of card sorting and 
semi-structured interviews. The results indicated that 
reading newspapers was one of their most common 


choices in public spaces. We also found that news was 


their favorite genre not only personally but also socially. 


Although newspapers were often used by caregivers 
as an evocative material in scheduled social activities, 
they rarely trigger residents’ communications in their 
daily lives due to the difficulties of describing and 
understanding between each other. Guided by these 
insights, we started with newspapers as an example to 


design a prototype system. 


3.2 Participatory Design Sessions 


The participatory design (PD) sessions were conducted 
with 8 residents to establish a preliminary concept. 

In each session, the designers acted as an impartial 
moderator to collect user requirements through active 
dialogues with residents. The topics of the dialogues 
focused on the participants' use of newspapers, their 


preferred sections, perceived attractiveness and 


Fig. 1.An overview of 
the preliminary concept 


design: R2S 


sociability of the combination of digital and print media 
[20]. We also brought newspapers and common smart 
devices such as tablets and smartphones to simulate 
future scenarios. We found the participants’ perceived 
attractiveness and sociability of augmented print 
media were positive. The priorities of their design 


requirements were concluded as follows: 


1) The interaction with print media should be simple, 
effortless and interesting. 

2) The interaction with digital information should be as 
simple as possible. 

3) The digital interface could be designed with familiar 
elements to older adults. 

4) The system should allow residents to freely decide 
the content and time to augment. 

5) Videos and images are the main forms of digital 
augmentation. 

6) The digital information should be continuously 
updated. 

7) The systems should be used by individuals or shared 
in small groups rather than the whole public area. 

8) Residents’ common interests such as local news are 


the primary contents to augment. 


4. Design of R2S 


4.1 Concept Design 


Although many design criteria derived from the PD 


sessions were still general and not enough to construct 


a completed system, we believe it is necessary to 
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explore an initial concept to be refined through the 
following user tests. Figure 1 provides an overview 

of the preliminary system design called ‘R2S’ (reading 
to sharing). It is a tabletop system distributed in the 
primary public area of care homes. Each unit of the 
system mainly consists of a tangible tool, newspapers 
enhanced with special marks (colored circles) and a 
digital display running R2S application. The marks can 
indicate the interactive areas on the newspapers. These 
areas are specially enhanced by related technologies 
for printed matter recognition. The tangible tool is a 
wireless tabletop device to identify each mark. It has 

a concise appearance with a white cylindrical body 

and a red switch at the top. It can stand steadily so 
that users do not need to keep holding it. Residents 
can decide when and which content to augment. They 
just need to place it on a certain mark to get access 

to corresponding digital content from the screen. The 
digital media could be directly played online from news 
websites or social media so that residents can check 
the updates. The digital interface is also very simple. 

It directly displays digital videos or images with brief 
descriptions in digital texts. When no one uses the 
system, the screens display nothing to avoid disturbing. 
As shown in Figure 1, residents can not only use R2S 
individually to compensate their physical barriers of 
reading newspapers but also share it to their social 
partners through the public display, which requires very 
little effort to describe and understand. 
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Fig. 2. The prototype 


implementation of R2S 


4.2 Prototype Implementation 


For rapid prototyping, we adopted NFC technology 

to bridge printed content and digital information. As 
shown in Figure 2, multiple transparent mini NFC tags 
are attached to newspapers. They are surrounded by 
sketched blue circles. The shell of the tangible tool is 3d 
printed with ABS. An NFC reader (RC522) is installed 
at the bottom of the shell to recognize nearby NFC 
tags. The tangible tool can be paired with a computer 
via Bluetooth, which is realized by a Bluetooth module 
(HC-05) connected to an Arduino Micro board. The 
board is powered by a lithium battery (7.4V) and also 

in charge of the data communication with the NFC 
reader and computer. The R2S software application 

is simulated with a live-programming platform VVVV 
that is characterized by real-time rendering and quick 
visualization. The identity code of each tag can be 

read by the tangible tool and sent to the computer via 
Bluetooth. The simulated application can receive the 
code, link them to related local or online media files and 


display. 


5. User Trial 


To initially collect user feedbacks and further refine 
R2S, a user test was conducted in the canteen of a 
care home belonging to the same caring organization 
mentioned above. According to the caregivers, the 


canteen was the main public area where most residents 


Fig. 3. [he demonstrated 
videos of existing solutions and 


the materials used in the user 


study. 


would like to stay. Five residents (2 males and 3 females, 
aged from 65 to 92) agreed to participate individually. 
They were recruited based on the same principle with 


the one in the design process. 


Each session was started with a brief verbal 
introduction of the design contexts and purposes. 
Then, we showed the participants six short videos of 
some existing solutions of augmenting paper interface 
to avoid restricting their minds within our design 
(Figure 3). The solutions were also printed as cards 

to facilitate recall and comparison. To test R2S, we 
firstly presented a 1-minute animated storyboard to 
provide an overview of the future scenarios. After this, 
we offered them the prototype to freely experience 
for 10 minutes and provide feedback. According to the 
residents' preferences reflected from the PD sessions, 
we enhanced the newspaper through four kinds of 
content including a piece of entertainment news, a 
piece of local news, real-time weather information and 
crossword puzzles. In the final step, we encouraged the 
participants to refine the prototype system or propose 
new solutions by quickly sketching on paper or digitally 
visualizing with the live programming environment 
(VVVV). We also prepared some related physical design 


references to facilitate this process (Figure 3). 


6. Result 


6.1 Concept Feedback 


All of the participants showed a positive attitude 
towards the idea to bridge print and digital media. 


However, none of them thought the demonstrated 
existing solutions could be applied in care homes. 

Even though we explained simultaneously, few of 

them could eventually understand the demonstrated 
user interactions that have been taken for granted by 
young people. P1 said: "My granddaughter taught me 
many times, but still cannot use it (tablet)." "They look 

too futuristic. People here may get curious, but most of 
them always keep a distance from the innovations." P5 
remarked. When showing the storyboard of R2S, all 
the participants could quickly understand the concept 
because it illustrated their daily lives. Overall, they were 
optimistic about the future application. P5 said: "Such 
thing is important to provide different things for people here 
to spend their time. Their life is too structured." He could 
already envision many social scenarios such as watching 
sports news and discuss local events together. They all 
agreed that the canteen was the ideal place to install 
the system for more social opportunities. P2, P3 and 
P4 emphasized the importance of distributed displays 
to entertain smaller social groups. In addition, all of 
them agreed that the tabletop tangible interface and 
interaction was much friendlier than the demonstrated 


modern digital devices and interactions. 


6.2 User Experience 


Although most participants needed our guidance to find 
the marks in the beginning, we found that they already 
learned how to use the system from the storyboard. 
When they got access to the digital augmentations, 
they firstly showed surprised and delighted with the 


experience. All of them could not help talking about 
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the detailed information on the screen, especially the 
local news. They praised the simplicity of the system. 
"Of course, it is very new, but it is not complicated. People 
will get familiar with it quickly." P1 said. Even though we 
encouraged them, no additional digital interface or 
interactions were required. “Select and watch. That’s it. 
The simple the better." P5 said. Although the tangible tool 
was very easy to use, some participants hoped it could 
be more attractive. “It should stand out among other daily 
items on the table otherwise people like me could not notice 


it." P2 suggested. 


6.3 User Refinement 


To refine the system, all the participants started with the 
tangible tool because it was what they directly interact 
with and they were more familiar with physical objects. 
They relied on the design references very much for 
inspiration. As shown in Figure 4, four of them thought 
the shape of the stamp was a better form because it 

is more interesting and more convenient to pick up or 
hold. P4 thought the pen was her favorite form because 
of better mobility, but she also admitted many residents 
here could not use pens for neuromuscular diseases. 
They all pointed out the importance of volume control 
because the acoustic environment in the public area was 
very unstable. "The volume needs to be loud if the group 

is watching it, but it may disturb others if it is too loud." P4 
said. Besides volume control, they were also encouraged 
to propose other potential functions, such as pausing, 
switching images, rewinding and zooming in (Figure 

4). However, they repeatedly emphasized that these 
functions were not necessary if they would make the 
system complicated. "You must keep it simple. Just basic 
functions otherwise people here will not use it." P2 said. In 


addition, most participants hoped the screen should be 
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Fig. 4. Five participants 
experienced the 
prototype and proposed 
ideas to refine R2S. 


a little larger for the better social experience and the 


marks on the paper could be highlighted with brighter 
colors or other materials. Their proposals on digital 

interfaces were very limited. Only P1 suggested there 
could be some simple digital instructions to guide new 


users. 


7. Discussion 


The participants' positive attitudes towards R2S 
identified a promising direction to design social 
technologies by augmenting their daily items or activities. 
However, their resistance to the demonstrated existing 
solutions also showed the necessity of certain design 
principles. R2S was designed following the criteria that 
we gathered from the PD sessions. Many requirements 
were further confirmed in this user trial. We believe 

this study could shed a light on designing socio-technical 
interventions in residential care settings. We discuss the 


design implications in the following three levels. 


7.1 Design for Accessibility 


The sociability of such technologies can hardly be 
realized if residents keep a distance. That might be 
the reason why the top three requirements from PD 
sessions focus on good accessibility. The simplicity 
and familiarity of the form, function and interaction 

is the premise for the residents to start to use the 
technologies. The feelings of technical intrusiveness 
should be minimized. In this case, tangible interfaces 
and interactions were preferred by the participants. 
All the participants refined the system into something 
they were familiar with such as stamp and pen. Their 
proposed function and interaction were inspired by 
their experience of using television and radio. Although 


various new features were proposed in the refinement, 


most participants were willing to sacrifice them to keep 


simplicity. 


7.2 Design for Attractiveness 


Based the first level, as the requirement No.4 indicated, 
the participants pursued better attractiveness to keep 
using the technologies. Besides the sense of control, 
this user trial identified other requirements related to 
pragmatic and hedonic qualities. In this level, designers 
should consider not only ergonomic factors to lower 
their physical barriers, but also aesthetic, psychological 
and emotional factors to increase their confidence and 
interest to use. In this case, the participants proposed 
various new forms, functions and interactions to meet 
their different needs. However, designers should 

not enrich the user experience by sacrificing the 
accessibility. We suggest the systems could be designed 
in a hierarchical structure. The basic functions should 
be embodied in simple forms and interactions for new 
users and the residents with lower capabilities. Some 
"hidden features" could also be provided for richer 


experience and long-term use. 


7.3 Design for Sociability 


We believe the sociability could be naturally achieved 
once the accessibility and attractiveness are properly 
addressed. In order to gather in-depth feedback and 
given the difficulties to simultaneously involve multiple 
older people to refine the system, this user trial was 
conducted with individuals rather than groups. Although 
the social functions of R2S could not be directly 
evaluated, from our observation and user feedback, 

we initially confirmed the requirement No.5, No.7 and 
No.8. The result indicated that the properties of shared 
displays (size, number, location, etc.) and the displayed 
content are two important factors that could contribute 
to the sociability, which needs to be further evaluated 

in group tests. We also believe the requirement No.6 is 
helpful to sustain residents' social interaction, but it also 


requires future field tests in longer terms. 
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with Chronic Stroke 


Abstract 


Stroke affects 15 million people worldwide and 

is the third most likely cause of death. Unilateral 
impairments after a stroke affect the ability to carry 
out daily activities with the affected arm and hand. 

The preferential use of the less affected side can lead 
to a so-called learned nonuse. Although a number of 
assistive everyday objects are available to people with 
stroke, little is known about how everyday objects 
could be designed to be useful for a self-directed 
rehabilitation and how the user experience of stroke 
patients could be evaluated to design complex digital 
and/or physical systems. We developed a set of 

design criteria that other designers can use to create 
rehabilitative objects. We interviewed 12 stroke health 
professionals to investigate what the user experience 
in the rehabilitation process looks like, and which 
clinical criteria need to be considered in the context 
of designing rehabilitative objects. Interviews were 
analysed using thematic analysis. Results indicate that 
to encourage the use of the affected arm and hand in 
people who live at home and have reached the chronic 
stage of the recovery, self-efficacy needs to be in place. 
Furthermore, people must undergo a behaviour change 


in order to overcome the learned nonuse. 


Keywords 


Stroke Rehabilitation, Qualitative, Design Criteria, 


Learned Nonuse 
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1. Introduction 


A stroke is a form of brain injury caused by lack of 
blood flow or oxygen delivery that leads to irreversible 
injuries to parts of the brain and can affect physiological 
as well as psychological abilities. A stroke affects 
approximately 15 million people annually worldwide, 
causing the deaths of one third, while another third 
survives with persistent disabilities [1]. Physical 
impairments commonly affect one side of the body in 
the form of hemiparesis, which is weakened muscles, or 
hemiplegia, which involves the paralysis of muscles [2]. 
Recovery from a stroke is a complex process that is 
likely caused by a learning-dependent process and a 
factor called spontaneous neurologic recovery [3]. The 
concept of spontaneous neurological recovery [4] is 
often referred to when explaining why improvements 
of motor impairments seem to plateau after the first 
weeks [5] in the upper limb within the first three 
months [6]. After this time, stroke survivors are likely 
to compensate for lost motor function of an arm with 
an enhanced movement of their less affected side, 
leading to a self-taught suppression of movement named 
the ‘learned nonuse’ of the affected arm [7]. Learned 
nonuse can be overcome by applying a physical restraint 
on the affected arm to evoke an initiation of use of the 
affected side while providing repetitive, contextualised 
training and behaviour change components. The 
intervention based on this principle is named constraint 


induced movement therapy (CIMT). 


Assistive technology [AT] is used after a stroke to 
compensate for the loss of functionality of the upper 
and lower limb and enable a safe discharge into the 
community. AT can be part of activities involving 
employment, mobility or communication [8, 9]. 
Commonly provided products after discharge from 
hospital are used for bathing, walking, and home 
adaptations such as modification of stairs [10—12]. 
Products and technology that fall into the category 
of assistive products help compensate for the loss of 
functionality rather than contribute to the rehabilitation 


process. 


While these objects may help stroke survivors carry out 
everyday activities again, we argue that they facilitate 
compensatory movement patterns and potentially 
‘learned nonuse’, as they help avoid the use of the 
affected limb. We believe that everyday objects could 

be designed to facilitate the use of the stroke patients' 
affected side, therefore contributing to their recovery. 
It has further been reported that people suffering from 
complex regional pain syndrome can express behaviour 
patterns similar to learned nonuse after a stroke [13]. 
Designing for initiation of use of the affected limb 

for stroke rehabilitation is a highly complex human 
experience and specific design strategies can help 
designers to design digital or physical systems for it [14]. 
Our design strategies could potentially be beneficial 

for other clinical populations as well. Also, at this stage 
it is unclear what kind of design criteria such objects 
need to fulfil. We have developed a set of design criteria 
that other designers can refer to in order to design 
objects for a society that is becoming unavoidably older 
and more likely to survive a stroke and be affected by 

it. We conducted 12 semi-structured interviews with 
healthcare professionals to gain an understanding of 
clinical factors influencing the user experience of such 
novel designs and to develop a set of design criteria that 
other designers can refer to in the process. This will 
hopefully help designers in this challenging and complex 
system that involves non-typical physical abilities, 


cognitive challenges, and intended behaviour change. 


2 Methods 


2.1 Semi-Structured Interview Content 


We used semi-structured interviews with stroke 


healthcare professionals to gain a deeper understanding 
of the rehabilitation process and strategies that can help 
to overcome learned nonuse. The primary questions 

in the interview guide were kept broad and open. 
Prompts that repeated the key concept were used 
during the interview to gain more in-depth information. 
The interview script outlined the main topics for 

the interview and started with questions about the 
participant's expertise. The flow of the interview 
influenced the order in which the remaining primary 
interview questions were asked. If participants did 

not elaborate on the concept of learned nonuse or 

had little involvement with chronic stroke survivors, 
additional follow-up questions were asked to ascertain 
if they were aware of the concept. Each interview was 
transcribed in full. For the interviews a set of main 


questions and follow-up questions were used: 


* How is the rehabilitation process structured? 

* Is the movement constrained during rehabilitation? If 
yes, in which form? 

* Which kind of emotions are influencing the 
rehabilitation process? 

* What kind of resources can people with stroke access? 


* How would the ideal rehabilitation system lool? 


Follow up questions concerned the following points: 

* How is feedback provided during the rehabilitation? 

* What are the steps to set up goals for the 
rehabilitation? 

* How are home-based assignments implemented in the 
process? 

* How do you try to evoke an initiation of use of the 


affected arm and hand? 


2.2 Recruitment 


We interviewed health professionals who had worked 
with acute, subacute or chronic stroke patients in 
clinical practice or research context. The interviews 
took between 30 and 120 minutes (average 60 minutes). 
Non-probability snowball sampling [15] was chosen for 
this study, in which participants recommended further 


possible interviewees. 


2.3 Thematic Analysis 


The transcripts were analysed using thematic analysis 


in a theoretical or deductive form to identify, report, 
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and analyse patterns and themes within the data [16]. 
Interviews were conducted by the first author. Initial 
codes and themes were generated by the first author 


and reviewed by the second and third authors. 


3. Results 


3.1 Study Sample 


Twelve interviews were conducted with stroke 
therapists and stroke researchers. Eight interview 
partners were female and four were male. Three 

were occupational therapists; two worked 

as physiotherapists; one worked as a neuro 
physiotherapist; one worked as a professional therapist 
focusing on assessment post-stroke; one worked in the 
tertiary sector and had a background in musculoskeletal 
therapy; two worked as researchers focusing on stroke 
therapy; one worked as a scientist focusing on the 
recruitment of acute stroke patients for studies; and 
one was a supervisor of a clinic. Two therapists worked 


in Australia and ten worked in New Zealand. 


3.2 Design Criteria 


Based on analysis of the interviews, we developed a set 
of design criteria [17] that can be used in the design 
process to create everyday objects with a rehabilitative 


purpose. 


3.3 Themes 


The following section outlines details of the themes 
that were mentioned during the interviews. The section 
has been divided into firstly, factors that facilitate 
compensatory movement patterns after the stroke 

and consequently a neglect of use of the affected arm 
and hand and, secondly, the structure of rehabilitation 
interventions including CIMT that aim to minimise 
compensatory movement patterns and evoke an 


initiation of use among people with stroke. 


Factors Contributing to ‘Learned Nonuse’. The 
development of compensatory movement patterns 
and learned nonuse after a stroke is multifactorial and 
develops over time; see Figure 1. Four different factors 
were mentioned during the interviews that contribute 
to its development after stroke: physiological factors, 
the structure of current rehabilitation, individual 


behaviour, and social factors. 
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Restrain The interaction with the objects needs 
to restrain movement to evoke an 
initiation of use and remind the user to 


use the affected arm 


Contextualise The object needs to be used within an 


activity of daily living 


Repetition The interaction with the object needs 
to be repetitive 

Feedback Feedback needs to be positive, 
immediate and quantitative 

Challenge The interaction with the object needs 


o become progressively challenging 


Self-efficacy The interaction with the object needs 
o contribute to developing self- 


perceived self-efficacy of the user 


Usability The interaction with the object needs 


o be achievable for a person with 


motor impairments after a stroke 


Behaviour change | A behaviour change component needs 
to encourage the use of the affected 


arm and hand in daily activities e.g., 


behaviour change contract 


Table 1. Design criteria 


Physiological factors included factors such as cognitive 
impairments; apraxia, which causes difficulties in 
understanding the purpose of the rehabilitation; 
spasticity; fatigue; visual neglect; a shift of body 
awareness; and sensory deficits that can cause issues 
with proprioception and light touch. Once a person 
starts to compensate with an enhanced use of the less 
affected side, the muscles on the affected side start to 
decondition, causing a decline in muscle strength and 


muscle bulk. 


What it tells us is that even a moderate level of impairment 
will promote nonuse. You have to be really, really good 
before you routinely use that hand and arm for everyday 
tasks and that is shocking. So the impairment scale is 66 
out of 66 and we have found, and others have found, that 
before that if your score falls below around 55 you won't use 
that hand and arm. Your MAL [Motor Activity Log] score 
will drop below three for both amount and quality of use. 
Scientist 01 (Participant 05) 


Fig. 2. Factors 
influencing the 
development of 
learned nonuse 
based on interview 


results. 


Injury e.g. stroke 


Depressed central nervous system and motor activity 


| 


INDIVIDUAL BEHAVIOUR SOCIAL FACTORS 


Forgetting to use the 
affected limb 


Family members and 
hospital staff tend to 
compensate for lost 


The use of adaptive meter slates 


technology to 
compensate for the 
affected side 


Faster performance 
with the less affected 
arm 


An affected non-domi- 
nant side is likely to be 
compensated for by 
the less affected side 


A focus in the acute and subacute rehabilitation process 
seems often to be on the lower limb, due to the stroke 
patients’ wish to be able to walk again. This is also 
reinforced by the target setting of health services to 
minimise inpatient stay getting patients ambulatory for 
discharge. One result is that focus on the upper limb 
occurs relatively late in the rehabilitation process or not 


present at all. 


It was further mentioned that people with a mild stroke 
and a rather high level of motor functionality are often 
confronted with statements telling them that they are 
not expected to get much better and that the focus is 
on more severe and acute patients. Historically, the 
focus on compensating for the loss of functionality 
occurs once a person has reached the chronic stage. 
The focus on compensation and use of assistive devices, 
often provided by occupational therapists, can further 
contribute to the development of learned nonuse by 


facilitating the use of the less affected side. 


"They basically tell people when you are on a higher level: 
‘There is nothing you can do’; ‘you are pretty good, so just 
be glad that you are as good as you are’; ‘go home and 
don't worry about the things that you would like to do’; ‘just 
thank god that you are alive and not too bad’. What is bad, 
l reckon. | think because | never ever met any stroke patient 
who told me that they are not too bad." 

Occupational therapist (Participant 02) 


PHYSIOLOGICAL FACTORS 


Visual neglect and sensory 
impairments impact on the 
performance 


THE STRUCTURE OF CURRENT 
REHABILITATION 


Ambition of the health 
services to minimise the 
inpatient stay 


Lack of information about 
how the movement 
should be performed 


Fatigue 


Decreasing muscle 


strength and muscle bulk 
Limited access to 


specialised stroke 


Restricted use (spasticity) —— 


Shift of body awareness Acute stroke therapy 


focuses often primarily 


Apraxia on the lower limb 


Limited access to stroke 
rehabilitation 6 months 4 


People with stroke tend to neglect the affected side 
after failed attempts during the acute stage. The self- 
taught compensatory movement is reinforced through 
the easier and faster performance of the less affected 
side and simply forgetting to use the affected arm. It was 
further mentioned that the effects of ‘learned nonuse' 
seem to be stronger in people with stroke who have 
reached a late chronic stage, that unsupervised training 
can reinforce compensatory movement patterns, and 
that there is an increased risk when the non-dominant 


hand is affected. 


"People quite often quickly become one-handed especially 
when it is their non-dominant hand that is affected by the 
stroke." Physiotherapist (Participant 06) 


Social factors contribute to the phenomenon as well. 
Relatives and occasionally hospital staff would try to 
make the life of the stroke survivors as easy as possible 


by doing tasks for them. 


Initiating the Use of the Affected Arm and 
Hand. In the following section the initial assessment, 
goal setting, feedback, emotions and elements of an 
effective rehabilitation intervention will be outlined. 
Figure 2 highlights which factors are part of a general 
rehabilitation right after the stroke. It additionally 
outlines themes mentioned in the context of learned 


nonuse. 
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ASSESSMENT 


Determine motor deficits 


Determine the amount of 
learned nonuse 


Therapist can understand 
motivations, daily routine 
and attitude of the 
person with stroke 


Baseline for outcome 
measurements that can 
be reflected back 


Essential elements in the rehabilitation process 


GOAL SETTING 


Often have given up trying 


to use the arm at all 


Goals need to be relevant 
to the person, specific, 
achievable, purposeful, 
part of daily activities and 
potentially competitive 


Milestones are defined 
that help during the 
process to validate 
improvements 


Complex movements can 
be broken down 


Demonstrate the 
correct movement 


Include behaviour change 
elements such as a 


FEEDBACK 


Provide positive & 
instant feedback 


Person needs to try to 
perform the movement 
before feedback is 
provided 


Improve movement 
patterns and be a 
motivational aid 


Oral feedback as well as 
'hands-on' 


Visual documentations 
e.g. video recording 
offers room for 
discussion as to how to 
improve the movement 


Increase motivation to use 


EMOTION 


Person has to develop a 
feeling of responsibility 
for their recovery process 


Sadness, fear, 
depression, grief, anger 
or frustration about the 
rehabilitation process 
and funding 


Have to accept the loss 
of their previous role 


Positive emotions were 
mainly associated 
positive rehabilitation 
outcomes 


Confronting situation 
when they have to work 
hard during the therapy 


CRUCIAL REHABILITATION 
ELEMENTS 


Further practice and 
reinforcement of using the 
affected arm 


Repeat the movement 


Contextualise the 
movement 


nclude family members in 
he process 


Provide information about 
he structure and purpose 
of the rehabilitation 


ncorporate self-directed 
raining elements in a daily 
routine 


Take account of the 
environment such as the 


behaviour change contract 


the affected arm 


person's home 


Increase self-efficacy 


The initial assessment of the impairments helps the 
therapist determine motor deficits, to understand 
motivations, daily routine and attitudes of the person 
with stroke to set appropriate recovery goals. The 
assessment is further used as a baseline for outcome 
measurements that can be reflected back to the patient. 
In the process different assessment scales are often 
used. If people had reached the chronic stage of the 
stroke and received rehabilitation interventions such 

as CIMT, the therapist would determine the amount 

of learned nonuse by using assessment scales such as 
the motor activity log (MAL), which is a 30-question 
interview guide to determine the amount and quality of 


daily use of the affected arm and hand. 


Based on the assessment outcomes, appropriate goals 
are determined. Goals that are focused on during 
rehabilitation need to be relevant to the patient, 
specific, achievable, purposeful, part of daily activities 
and potentially competitive. The goals should be task- 
based, rather than abstract ones such as '| want to use 
my arm again’. It was further mentioned that they need 


to be contextualised. 
The use of feasible, meaningful and specific goals for 
the process provides motivation in the often frustrating 


rehabilitation process. The person has to develop a 
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feeling of responsibility for their process. Once the 
goals are determined, milestones are defined that help 
during the process to validate improvements. Complex 
movements can be broken down into individual 
components to then be mastered and learned as a 
sequence. Demonstrating the correct movement helps 
the patient understand how the movement needs to 
be performed effectively. Repetition of the correct 
movement is a core element of the process. Video 
recording is used to provide visual documentation 

and offer room for discussion on how to improve the 


movement. 


The role of the therapist was described as a form of 
personal trainer that helps to reach the rehabilitation 
goals. They would provide motivational input in the 
form of feedback during the rehabilitation. Feedback is 
provided orally as well as in a 'hands-on' form. It should 
be provided instantly, appropriate for the person and 
task. Feedback is used to improve movement patterns 


as well as being a motivational aid. 


So you want people to remain as positive and optimistic as 
possible but you have to be realistic so [...] it just needs to 
be realistic so saying something like ‘Okay... | really see that 
you want get back to work but you are still in hospital right 


now’. So what steps do we need to take to get you there? 


Fig. 3. Elements 

to initiate the use 

of the affected arm 
in clinical practice. 
Learned nonuse- 
specific elements are 


highlighted in blue. 


"You need to do x, y, and z but right now you just need to 
focus on the first part. To be able to get back to work you 
might need to be able to walk or you might need to be able 
to concentrate to complete a task for an hour or so." 


Professional therapist (Participant 04) 


In order to evoke an initiation of use of the upper limb 
in people with signs of learned nonuse the therapist has 
to provide sufficient motivational input and feedback 
to convince the patient that they should be trying to 
use the arm and that it is just the learned nonuse that 
impacts on their functional performance. When people 
with chronic stroke go back to see a therapist, they 

are often depressed and have given up trying to use the 
affected arm at all. When they are asked to work hard 
during the therapy session it can be experienced as an 


extremely confronting situation. 


"You can't start these guys off trying to do more complicated 
things because they are doing nothing. So even if you think 
that they have some capacity [...]. The first couple of days 
with those people are always focused on trying to get them 
to just initiate the use of it and that can be anything really." 
Occupational therapist (Participant 02) 


The emotions that the patient goes through can have 

a significant impact on the process. Most therapist 
mentioned mainly negative emotions such as sadness, 
fear, depression, grief, anger or frustration about the 
rehabilitation process and funding system. Patients must 
go through the process of accepting the loss of their 
previous role such as being the main provider for the 
family and accepting their new role and limitations post 
stroke. Rarely would someone see the stroke as a life- 


changing event that transforms their life positively. 


"We even have patients crying in the gym out of frustration 
because they are disappointed." 


Supervisor of a student-run clinic (Participant 09) 


Positive emotions were mainly associated with positive 
rehabilitation outcomes. The feeling of surprise after 
being able to perform movements was mentioned as 
evoking the wish to continue trying to use the affected 
arm in more activities once positive results have set in. 
It was emphasised that to overcome the signs of learned 


nonuse the person needs to go through behavioural 


changes. To secure an effective treatment, carers and 
family members are informed about and involved in the 
process. The use of a behaviour contract signed by the 
therapist and the person with stroke seems to be an 
effective tool to evoke a feeling of mutual responsibility. 
There are different techniques such as diaries or home 
practice sheets that help remind the patient to use the 


affected arm as often as possible. 


The use of a physical constraint as part of CIMT to 
evoke an initiation of use was mentioned by a small 
number of our interview participants. The functional 
purpose is not primarily to restrict the movement, but 
to remind the patient to use the affected arm and limit 
the possibility of compensating with an enhanced use 
of the less affected side. It was emphasised that CIMT, 
which uses a physical constraint as part of the protocol, 
is a behaviour change intervention and that the physical 


constraint is just one element of the intervention. 


4. Discussion 


Assistive devices such as one-handed chopping boards 
and stroke cutlery are often provided to people who 
have experienced a stroke. These devices help to 
compensate for the lost motor abilities rather than 
contributing to the rehabilitation process once the 
person with stroke has returned home. However, 
devices such as the one-handed chopping board do 
not restrain movement of the affected arm and hand 
to contribute to an initiation of use, but enable the 
user to chop food one-handed. Furthermore, it does 
not contribute to a behaviour change to make sure 
that gains made in the process persist in the long- 
term. These are qualities commonly found in current 
rehabilitation devices. They demonstrate the lack 

of guidelines for designers to develop products and 
systems that properly address and evaluate the needs 
and complex situations that stroke patients face. 

This is not surprising, as designers also experience 
complexity when designing for stroke patients. Our 
findings will hopefully assist designers in this process by 
contributing to the body of knowledge in the discipline 
through strategies and concrete criteria. We conducted 
12 interviews with stroke rehabilitation health 
professionals to develop an understanding of the user 
experience of people with stroke from multiple clinical 


perspectives. We developed a set of design criteria that 
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can be used to develop objects with a rehabilitative 
character. Our results indicate that there is little focus 
on people who have reached the chronic stage of the 
stroke recovery and, due to financial and other resource 
limitations, the focus of current rehabilitation is often 
on the lower limb. It is therefore unclear if people who 
reach the chronic stage of the stroke have reached their 
full potential for recovery of use of their upper limb, or 
have started to express signs of learned nonuse. People 
with stroke have to face a constant struggle of accepting 
the new situation or waiting for improvements. Studies 
indicate that change can be experienced as necessary 
but associated at the same time with abandoning 
possible improvements by relying on technical aids, 


environmental improvements and other people [18]. 


Our study indicates several factors that contribute 

to the development of learned nonuse in people with 
stroke. The descriptions of physiological factors and 
individual behaviour contributing to learned nonuse 
correlate with its description in the literature as a 
self-taught behaviour that develops over time and 

is reinforced through unsuccessful motor attempts, 
failure, punishment and pain [19]. Additional personal 
and physiological factors mentioned in previous studies 
are: the extensive time to plan, initiate and complete a 
movement; fatigue; lack of confidence and control over 
the situation; pain; fear of negative consequences; and 


fear of becoming dependent on others [18,20]. 


The findings of this study in terms of social factors 
correlate with previous findings [21]. Additional factors 
are social isolation caused by losing contact with friends 
and colleagues; not belonging to the work community; 
and attempts to hide the condition in order to appear 
‘normal’ to others [20-22]. Another potential factor is 
gender. A study by Taub et al. [23] showed that female 
participants made greater gains on the 'Motor Activity 
Log' than males, with the possible explanation that 

the females were faced with a stronger expectation 

to use the affected arm to perform tasks again that 

are connected to traditional female roles such as 


housework and cooking. 
Initiating the use of the affected arm and hand. 


The initiation of use of the affected arm and hand in the 


chronic stage of the stroke is crucial to overcoming the 
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learned nonuse and decreased functional impairments. 
In the context of neuroplasticity, learned nonuse has 
been connected to the principle of ‘use it or lose it’ 
[24,25], meaning that once compensatory movement 
patterns facilitate a neglect of the affected arm and hand 
potential underlying abilities to use the arm are lost. 
CIMT offers a detailed intervention protocol to 
overcome learned nonuse and has been part of our 
interview protocol. Few participants in the current 
study were familiar with the protocol. Participants 
who used CIMT mentioned that the protective safety 
mitt was preferred over the sling or cast to restrain 
movement and initiate use of the affected arm and 
hand. The mitt minimises current criticism regarding 
the patient's safety by only limiting the ability to use 
the fingers and hand in activities of daily living, but 
maintaining the ability to extend the arm in the case of 
a fall or to compensate for balance problems [26,27]. 
Therapists who used CIMT emphasised that the 
intervention aims to evoke a behaviour change rather 
than being repetitive task training and the constraint is 
not the main component, despite the name suggesting 
this [28]. Findings of this study show that the physical 
restraint reminds the patient of an extrinsic motivator 


[25] rather than limiting movement. 


A number of factors were mentioned that seem to be 
unique in terms of initiating the use of the affected arm 
and hand and overcoming learned nonuse in the chronic 
stage of stroke. These are: assessment of the amount of 
learned nonuse; strategies to evoke the initiation of use 
based on assessed motor capabilities and motivational 
input; and the need to elicit a behaviour change. Feasible 
goals build the foundation to address learned nonuse 
successfully. The person with stroke is able to achieve 
the self-determined goals which improves self-efficacy 
[25]. Self-efficacy is the perceived competence to attain 
a specific level of performance in a given environment. 
It is the subjective cognition to carry out a particular 
behaviour and does not necessarily relate to the actual 
competence [29]. It seems to be essential to develop 
self-efficacy in order to choose to use the affected 

limb. However, little is known about it in the context of 
upper limb stroke rehabilitation due to a strong focus 


on the lower limb in most studies [25]. 


The mentioned elements of rehabilitation can be 


mapped to the elements of self-efficacy of mastery 
experiences, vicarious experiences, and verbal 
persuasion. However, the fourth source of self-efficacy, 
the physiological and emotional state [30], was not 
mentioned. The motivational input and keeping the 
patient engaged in the process of initiating use of 

the affected arm and hand was mentioned as a core 
element. Motivational input aims to convince the stroke 
survivor that they have the potential to use the arm and 
that it is just the self-taught behaviour that is stopping 


them from using it. 


To address learned nonuse and evoke initiation of use, 
the rehabilitative object needs to address and increase 
self-efficacy beliefs. However, this is a complex process 
involving multiple factors. Self-efficacy beliefs rely on 
acquirable skills, increasing the belief of the individual 
that they can gain these skills, modelling the requisite 
skills, and defining activities in a way that make them 
achievable, as well as providing explicit feedback during 
the process [30]. A behaviour change component can 
help to secure gains made in the rehabilitation process 


in the long-term. 


5. Limitation 


The interview results need to be considered in light of 
the study sample. Two participants worked in Australia 
where the health system and stroke guidelines might be 
slightly different. Furthermore, most of the participants 
were located close to Wellington and Auckland which 
are two of New Zealand's largest cities. Access to 
stroke services might be different in provincial areas of 
the country and indeed internationally. Since none of 
the stroke therapists were located in a rural area, there 
might be a potential bias included in the results. We 
invited participants working in the private and public 
health sector. However, the majority of participants 
worked in private clinics and research contexts where 
people with stroke might be more encouraged to work 
on their motor deficits. A closer collaboration with 
local hospitals and hospital staff might potentially help 
to recruit participants who work with people with 


stroke who are less motivated. 


It needs to be considered that the strategies focus on 
objects that can be used in the home environment as 


part of the rehabilitation process. Home-based stroke 


interventions were mentioned during the interviews 

as being a crucial element in the process. Nonetheless, 
there seems to be no consensus on one specific therapy 
approach. Furthermore, in New Zealand, access to 
stroke services for people with chronic stroke and 

mild impairments seems to be quite limited. One of 

the therapists mentioned that only highly motivated 
survivors with sufficient funding would try to improve 
motor impairments in the chronic stage, which indicates 


a lack of support for this stroke population. 


People with stroke were not involved in this study and 
they could provide further insight about additional 
design criteria that influence the use of rehabilitative 
objects in the chronic stage of the stroke. We have 
taken account of this and applied for ethics with the 
Health and Disability Ethics Committees in New 
Zealand in order to involve people with stroke in the 


second stage of our research. 


References 


1. McKay, J., Mensah, GA. (2004). The atlas of heart disease 
and stroke. Geneva: World Health Organization. 

2. Colman A.M. (2009). A dictionary of psychology [Internet]. 
Oxford: Oxford University Press. Retrieved Nov 1 2016, 
from: http://www.oxfordreference.com/views/BOOK_ 
SEARCH.html?book-t87 

3. Langhorne, P., Bernhardt, J., Kwakkel, G. (2011). Stroke 
rehabilitation. The Lancet. 377(9778). 

4. Gresham, GE. (1986). Stroke outcome research. Stroke. 17(3), 
358-60. 

5. Kwakkel, G., Kollen, B., Lindeman, E. (2004). Understanding 
the pattern of functional recovery after stroke: Facts and 
theories. Restorative Neurology and Neuroscience. 22(3—5), 
281-99. 

6. Zarahn, E., Alon, L., Ryan, S.L., Lazar, R.M., Vry, M-S., Weiller, 
C., et al. (2011). Prediction of motor recovery using initial 
impairment and fMRI 48 h Poststroke. Cerebral Cortex. 
21(12), 2712-21. 

7. Taub, E., Uswatte, G., Mark, V.W., Morris, D.M.M. (2006). The 
learned nonuse phenomenon: Implications for rehabilitation. 
Eura Medicophysica. 42(3), 241-56. 

8. Algurén, B., Lundgren-Nilsson, A., Sunnerhagen, K.S. (2009) 
Facilitators and barriers of stroke survivors in the early post- 
stroke phase. Disability and Rehabilitation. 31(19), 1584-91. 

9. ICF Research Branch. (2013). Comprehensive ICF core set 


Design and Semantics of Form and Movement — 175 


10. 


11. 


12. 


13. 


14. 


15. 


16. 


17. 


18. 


T9: 


20. 


21. 


22: 


176 


for stroke. Retrieved Jul 13 2017, from: https://www.icf- 
research-branch.org/download/send/12-cardiovascularandres 
piratoryconditions/199-comprehensive-icf-core-set-stroke 
de Barros, A.C., Duarte, C., Cruz, J.B. (2009). Designers and 
stroke research. Australasian Medical Journal. 10(1), 104—11. 
Gosman-Hedstróm, G., Claesson, L., Blomstrand, C. 

(2002). Assistive devices in elderly people after stroke: A 
longitudinal, randomized study - The Góteborg 70+ Stroke 
Study. Scandinavian Journal of Occupational Therapy. 9(3), 
109—18. 

Sumathipala, K., Radcliffe, E., Sadler, E., Wolfe, C.D., McKevitt, 
C. (2012). Identifying the long-term needs of stroke 
survivors using the International Classification of Functioning, 
Disability and Health. Chronic Illness. 8(1), 31—44. 

Punt, D.T., Cooper L., Hey, M., Johnson, M.I. (2013). Neglect- 
like symptoms in complex regional pain syndrome: Learned 
nonuse by another name?. Pain. 154(2), 200-3. 

Lemke, M., Rodríguez Ramírez, E., Robinson, B. (2017). 
Limited use only. How can the design of an everyday object 
help chronic stroke patients to overcome the learned 
nonuse of the upper limb?. The Design Journal. 28;20(sup1), 
8923972417. 

Atkinson, R., Flint, J., (2011). Snowball Sampling. In: Lewis- 
Beck M, Bryman A, Futing Liao T, editors. The SAGE 
Encyclopedia of Social Science Research Methods. Thousand 
Oaks: Sage Publications, Inc. Retrieved Mar 23 2017 

from: http://methods.sagepub.com/reference/the-sage- 
encyclopedia-of-social-science-research-methods/n931.xml 
Braun, V., Clarke, V. (2006). Using thematic analysis in 
psychology. Qualitative Research in Psychology. 3(2), 77—101. 
Rodríguez Ramírez, E. (2017). A postgraduate thesis model 
for research through design based on design criteria. The 
International Journal of Designed Objects. 11(4), 11-27. 
Wallenbert, l., Jonsson, H. (2005). Waiting to get better: A 
dilemma regarding habits in daily occupations after stroke. 
Therapy. 59(2), 218-24. 
Taub, E., Morris, D.M. (2001). Constraint-induced movement 


American Journal Occupational 


herapy to enhance recovery after stroke. Current 
Atherosclerosis Reports. 3(4), 279-86. 
Taule, T., Ráheim, M. (2014). Life changed existentially: A 


qualitative study of experiences at 6—8 months after mild 
stroke. Disability and Rehabilitation. 36(25), 2107-19. 
Bandura, A. (1997). Self-efficacy: The exercise of control. 
New York: W.H. Freeman. 

Warner, G., Packer, T., Villeneuve, M., Audulv, A., Versnel, 
J. (2015). A systematic review of the effectiveness of 


stroke self-management programs for improving function 


Design and Semantics of Form and Movement 


23. 


24. 


25. 


26. 


27. 


28. 


29. 


30. 


and participation outcomes: Self-management programs 
for stroke survivors. Disability and Rehabilitation. 37(23), 
2141—63. 

Taub, E., Uswatte, G., Mark, V.W., Morris, D.M.M. (2006). 


The learned nonuse phenomenon: Implications for 
rehabilitation. Eura Medicophysica. 42(3), 241—56. 

Cobley, C.S., Fisher, R.J., Chouliara, N., Kerr, M., Walker, M.F. 
(2013). A qualitative study exploring patients' and carers' 
experiences of early supported discharge services after 
stroke. Clinical Rehabilitation. 27(8), 750—7. 

Dancause, N., Nadeau, S., Rossignol, S. (2015). Sensorimotor 
rehabilitation: At the crossroads of basic and clinical sciences. 
Amsterdam: Elsevier. 

Taub, E., Crago, J.E., Uswatte, G. (1998). Constraint-induced 
movement therapy: A new approach to treatment in physical 
rehabilitation. Rehabilitation Psychology. 43(2):152—70. 
Uswatte, G., Taub, E., Morris, D., Barman, J., Crago, J. (2006). 
Contribution of the shaping and restraint components 

of constraint-induced movement therapy to treatment 
outcome. NeuroRehabilitation. 21(2), 147—56. 

Taub, E. (20 
induced movement therapy: An example of behavioral 
itation. Behavioural Analysis. 35(2), 155—78. 
Wood, J.P., Connelly, D.M., Maly, M.R. (2010). "Getting 


2). The behavior-analytic origins of constraint- 


neurorehabi 


back to real living": A qualitative study of the process of 
community reintegration after stroke. Clinical Rehabilitation. 
24(11), 1045—56. 

Kleim, J.A., Jones, T.A. (2008). Principles of experience- 
dependent neural plasticity: Implications for rehabilitation 
after brain damage. Journal of Speech Language and Hearing 


Research. 51(1), 225-239. 


Madeleine J. Hazelton, Gillian M. McCarthy, Edgar R. Rodríguez Ramírez 


Victoria University of Wellington 


madeleine.hazelton@gmail.com 


The Semantics of Conspicuity: Design 
Strategies to Address Conspicuity in Type 1 
Diabetes Medical Devices for Adolescents 


Abstract 


This paper reports on the development of semantic 
strategies designers can use when designing medical 
devices for adolescents with type 1 diabetes to match 
their preferences for conspicuity. Adolescents with type 
1 diabetes use medical devices that can cause unwanted 
attention in public settings and can create perceived 
social pressures. The semantic strategies address 
conspicuity by enhancing the traditional medical device 
to make it more beautiful, disguising the medical device 
as a non-medical item, concealing the medical device 
from the public, personalising the medical device for 
the user, and using technology to advance the medical 
device. Participatory design methods were used to 
understand adolescents' experiences of using medical 
devices in public and how the semantic strategies could 
be applied to match their preferences for conspicuous 
or inconspicuous medical devices. Both adolescent 
participants with type 1 diabetes felt comfortable 
testing in public; there were occasions where they 

did not want to answer questions from onlookers. 
Participants favoured a combination of the semantic 
strategies. In particular, they were making the device 
look less "medical" and more beautiful, and utilising 
technology to make the device more inconspicuous. 
This research developed semantic strategies to design 
new medical devices that match adolescent preferences 


for conspicuity. 


Keywords 


Type 1 Diabetes, Conspicuity, Semantics, Participatory 


Design, Adolescents 


1. Introduction 


Type 1 diabetes is an invisible, chronic autoimmune 
disease where the pancreas is unable to produce insulin 
to regulate glucose levels in the body. Adolescents 

aged 13-18 with type 1 diabetes are required to 

follow an intensive self-management regimen that 
includes, frequent monitoring of blood glucose levels, 
diet, exercise and self-administering insulin [1]. 
Adolescent development research shows that self- 
managing in public spaces can be challenging, as it 

can draw unwanted attention and questions, at a life 
stage when social acceptance is critical [1]—[3], this 

is commonly referred to as stigma. The term stigma 
did not accurately represent adolescent experiences, 
as it is a word that elicits strong emotions that not 

all adolescents feel towards their medical devices [2]. 
Whereas the term conspicuity does, as adolescents do 
not always want to be the centre of attention and would 


prefer an inconspicuous device in specific settings. 


Previously the priorities when designing for medical 
conditions and disabilities have been to enable the user 
while attracting as little attention as possible [4]. This 
traditional approach has been less about creating a 
positive image around a medical condition, and more 


about designing an object which projects no image at all. 


Semantics should be considered while designing 
medical devices [5] for adolescents with type 1 
diabetes. Krippendorf and Butter [6] have defined 
product semantics as: "the study of symbolic qualities 
of man-made forms in the context of their use and 


the application of this knowledge to industrial design." 
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Product semantics requires the designer to understand 
the context in which the product is used and addresses 
how the product communicates in its environment [6]. 
Semantics has been used in past research to describe 
strategies to elicit surprise through the design of a 
product [7], similar semantic strategies have been 
developed in this research; however, the strategies aim 


to design conspicuous or inconspicuous medical devices. 


Adolescents' self-management of type 1 diabetes should 
be considered in a social context, and the semantics of 
the medical device should be reflected when designing 
for adolescents with type 1 diabetes. In this research, 
we aim to develop strategies that address the semantics 
of conspicuity to enable designers to create medical 
devices that provide adolescents with the opportunity 


to disclose their medical condition when they choose. 


2. Methods 


2.1 Development of Semantic Strategies 


The strategies were developed by analysing literature 
through thematic analysis [8]. We used the six-phase 
approach to thematic analysis [8]. The thematic analysis 
comprised stages of familiarising yourself with the 

data, generating initial codes, searching for themes, 
reviewing potential themes, defining and naming 
themes and producing a report. The literature analysed 
included five different pieces that discuss strategies to 
reduce product-based stigma (being the more common 
terminology than conspicuity). The strategies in the 
literature were disregarded if they did not address the 
design of a tangible product. Table 1 describes the final 


themed strategies. 


2.2 Participatory Design 


Participants were two adolescents with type 1 diabetes 
aged 15. One boy and one girl took part in the semi- 
structured interviews, collage, and sketching. The two 
participants were involved in the initial development 
and exploration of the strategies. Participatory design 
methods were used in the research so that adolescents 
with type 1 diabetes were able to discuss the semantic 
strategies and share their opinions and experiences. 
Participation in design involves open dialogue, 
communication and trust [9]. Participatory design is 


a process of investigating, understanding, reflecting, 
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establishing and developing ideas with participants [10]. 


The participants took part in semi-structured 
interviews, collaging and sketching activities to discuss 
their preferences towards conspicuity and to visualise 
their preferred semantic strategies. A collage is a 

less formal and more creative way to discuss ideas. It 
provides adolescents with an opportunity to project 
personal information onto visual artefacts [11]. The 
images used for the collage are of products that could 
be categorised into one of the developed semantic 


strategies. 


Sketching with the adolescents' allowed the participants 
to sketch their ideal future concepts. The sketch 
concepts were influenced by the collage and the ideas 
discussed in the interviews. This is an opportunity 

for the adolescents to tailor the concepts towards 

type 1 diabetes medical devices and sketch new more 
conspicuous or inconspicuous devices based on their 
preferences. Thematic analysis was also used to analyse 
the interviews collages, and sketches. Themes from 

the interviews were identified and compared with the 


developed semantic strategies. 


3. Results 


The final results discussed in this section includes the 
five semantic strategies, the results from the interviews 
discussing the strategies, or experiences relating to 

the strategies and the participant sketches of possible 


medical devices. 


3.1 Semantic Strategies that Address the 
Conspicuity of Medical Devices 


The final semantic strategies developed through 
thematic analysis, range on a scale (fig. 1) from 
conspicuous to inconspicuous to allow for adolescents' 
preferences to be met. Strategy 5 and 4 could be 

both be designed as inconspicuous or conspicuous 
objects, therefore their points extend across the 

scale. Each strategy also provides a different level of 
consumer appeal and medical appeal, which can also 

be demonstrated in figure 1. The majority of current 
medical devices (fig. 5) would lie within the medical and 
conspicuous area of the diagram. The final strategies are 


outlined below. 


Table. 1. Strategies 
developed through 


thematic analysis 


Enhance the Aesthetic Appearance of the 
Traditional Medical Device. The advantages of 
enhancing the aesthetics of a medical device are that it 
allows the device to appear less "medical." By enhancing 
the aesthetics, it can make the device more innovative, 
stylish and modern. This can be achieved through 
colour, texture, pattern, material or form. An example 


of enhancing the aesthetic appearance of the traditional 


medical device is Epipi designed by Harry Moorman (fig. 
2). Epipi still retains some of the traditional elements 
of an EpiPen; however, it has been improved through 


choices such as material, colour, and texture. 


Personalise the Medical Device for the User. 
By personalising an object, it provides the user with 


a sense of ownership over the object. Personalisation 


[Theme | Source [strategy | Description of strategy 


Enhance the Fading out and Disguising 
traditional 


a medical device to Drawing Positive Attention 


make it more beautiful 


Reinforce the medical role 


m 
[v] 
c 


played by the device 


[13] Diversion of attention 


Disguise the medical [13] 
device as a non- [13] 


medical item 
[13] Reshaping product meaning 


hrough meaningful interaction 


with other products 


Downplay stigma sensitive 


features by disguising the 
product as an accepted non- 


medical item 


Conceal the medical [14] 


device from the public [3] Minimise stigma-sensitive 


aspects of the device by making 


it invisible or less confronting 


and minimise its presence. 


Hide and replace unpleasant features, with more pleasant ones, 
which fit the context. 

Design devices that create positive attention for assistive 
technologies 

Design the device with materials, colours and textures that make 
it appear medical. 

Create a diversion of attention away from the medical aspects of 


the device. 


Camouflage — disguise Camouflage or hide the medical aspects. 
The device provides another ability. 


[he interaction of products can be cooperative, competitive, or 


independent. 


Make the device less noticeable and allow the user to hide their 


type 1 diabetes. 


Covering the stigmatising object | Conceal the disabling or stigmatising part of the product 


Make the device less noticeable and less embarrassing in public 


[12] Hiding and Covering Hiding to manage both visible and non-visible stigma. 


Personalise the [12] 


medical device for [11] Individual identity 


the user 
[3] Strengthen the individual 

identity of the device by 
providing more opportunities 
for personalisation, so users 
value it as an extension of their 
personality. 

Use Designing Stigma Out 

technology to advance 

the medical device 


Reshaping product meaning 


through advances in material 


technology 


Products feel more tailored to the user 


Allow the user to personalise their device, either through mass 
customisation, or the user incorporates their own identity. 
Provides participants with a sense of control and ownership over 


their device. It reflects the user's style and allows them to feel 


proud of their device. 


Technology can be used for reshaping the user. Assistive products 
can become more varied, tailored and personalised. 
By integrating new technology into an existing product, it provides 


the designer with a way to reshape a product 
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Enhance the aesthetics of the 
traditional medical device 


Disguise the medical device 
as a non-medical item 


Conceal the medical device 


from the public Inconspicuous 


Personalise the medical 
device for the user 


Use technology to advance 
the medical device 


can be achieved through providing the user colour 
options, patterns and engraving. Jacobsen [12], Vaes 
[13] and McCarthy [3] all mentioned this strategy in 
their research as a way of making the user proud of 
the object and therefore encouraging them to use their 


devices more regularly. 


The Hue inhaler (fig. 3) designed by Tim Zarki is 

a device that demonstrates the opportunity for 
personalisation. The Hue inhaler is a device that not 
only fits into the personalisation strategy but also could 
be an example of enhancing the aesthetic appearance 
of the traditional medical device. Personalisation is a 
strategy that aligns well with several other strategies, as 
it can be achieved through small changes, and provides 


the user ownership over their medical devices. 


Disguise as an Accepted Non-Medical Device. 
This strategy aims to remove the ‘medical’ aspects from 
the design to create a conspicuous object that can be 
used openly in public spaces. Disguising the medical 
device as another object allows adolescents to use the 
product in public, without the object being recognised 
as a type 1 diabetes medical device. This strategy is 
similar to, two of Vaes's [13] strategies that aim to 
reduce stigma in products. The strategies aimed to 
camouflage or disguise the products. McCarthy [3] also 
used this strategy; however, renamed it as "disguise the 
product as an accepted non-medical item." Examples of 
this strategy could include disguising medical devices in 
everyday objects such as a watch, cell phone, or keys. 
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Fig. 1. Diagram showing the 


Medical 


appeal of the strategy 


Conspicuous 


Jauinsuo) 


Conceal the Medical Device from the Public. For 
adolescents who are more embarrassed by their type 1 
diabetes or their medical devices, they might gravitate 
towards concealing their medical device from the public. 
This strategy is different from disguising as an accepted 
non-medical device, as it asks the designer to remove 
the object from the view of the public altogether. 

This strategy often works well with technological 
advancement, as many of the opportunities to design 
for this strategy are through wearable technologies 
such as temporary tattoos (fig. 4), contact lenses, and 
microchips. This strategy is a useful one to consider 

as it withdraws the device from the view of the public, 
making it more comfortable for adolescents to test in 


public discreetly if they choose to. 


Technological Advancements. As technology 


advances, the desire for newer, smaller and slimmer 


type 1 diabetes medical devices is becoming apparent 


conspicuousness and consumer 


Fig. 2. Epipi. Image 
courtesy of Harry 


Moorman. 


Fig. 3. Hue Inhaler. 
Image courtesy of Tim 


Zarki. 


Fig. 5. Current 
traditional type 1 
diabetes medical 
devices available in New 


Zealand 


[15]. Developments in technology have allowed for 


more experimental blood-glucose meters such as the 
Freestyle Libre, the tattoo blood glucose sensor, the 
Google lens blood glucose meter, and blood-glucose 
ink. As technology develops, it will become increasingly 
more accessible for us to visualise the possibilities for 


new, more inconspicuous medical devices. 


3.2. Conspicuity of Testing in Public 


In the interviews, both participants were not concerned 
about testing in public; participant 2 responded: “I don't 
mind testing in public, because | feel like it’s something you 
need to do." It is important to respect the adolescents’ 
experiences; however, there is a nuance to attention. 
Some adolescents might not feel embarrassed but do 
not want to be forced to disclose their type 1 diabetes 
or made to answer questions at that moment. We 
recognise that there is a variety of opinions, and the 
preferences of other adolescents' that have not been 


interviewed should be considered. 


The adolescents interviewed were also aware of this; 
participant 1 had been diagnosed at a young age but was 


aware that someone who had been recently diagnosed 


might have different opinions about testing in public. 
“I’ve had it for 11 years, but if someone had it for three 
months or something, it might become more awkward for 
them, and they might want to go off and do it privately." 
Therefore, it is essential to consider the occasions when 
the participants do not want to be asked questions 
about their medical devices or type 1 diabetes, as they 


might be busy, not in the mood or embarrassed. 


One of participant 1's first thoughts about creating a 
conspicuous device was: “I think the whole thing of this is 
trying to make sure people do not look at it and immediately 
think ‘oh it's medical, what's wrong?" Participant 2 was 
less concerned about the device being medical and more 
interested in making the device "innovative and stylish." 
Current devices (fig. 5) available in New Zealand use 
colours, materials, shapes and interfaces that make the 


device appear traditionally ‘medical’. 


Type 1 diabetes devices are frequently a blue colour, as 
participant 2 explained: “Well you know how there is a 
blue theme going through it, because it’s clean, but it doesn’t 
need to be because people don’t want all blue.” Participant 
1 also mentioned that “there is like six colours for the 
pump, but it’s very basic navy blue, green, black and grey.” 
Although participants are not concerned about testing 
in public, they still desire a device that provides them 
with the autonomy to disclose their type 1 diabetes 
when they chose. A combination of the semantic 
strategies would provide them with a device that 


matches their preferences for conspicuity. 


3.3. Preferences for Semantic Strategies 


The participants did not gravitate towards a specific 
strategy in their interview, collage or sketches. Both 
participants preferred a combination of strategies. Two 
of the themes generated from the interviews involved 
technology and aesthetics; these were the preferred 
strategies from the participants combined with aspects 
from other strategies. Several ideas were discussed 
that generated ideas which combined aspects of the 


strategies. 


Preferences for Strategies as Discussed 
in Interviews. Participant 1 was interested in 
exploring the possibilities of either making the 


device inconspicuous and not seen by the public or 
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conspicuous and onlookers know what the device 

is. Participant 1 believed when a medical device is 
conspicuous and appears medical such as a wheelchair, 
crutches or even a tracheostomy; onlookers typically do 
not ask questions as they are aware of what the item is. 
Questions are also not asked when the onlooker cannot 
see an item at all. People will begin to ask questions 


when they are unsure if the object is medical or not. 


P1: I think the main thing is that there is something where 
it's so obvious it's not medical- where people don't ask about 
it. Then there is a period where it shows that it's medical, so 
people don't ask about it, and then there's that middle zone, 
where people are unsure whether it's medical or not, so 
people ask you about it. But if you make it really discreet or 
really obvious people won't ask you about it. 


One of the ideas Participant 1 discussed to demonstrate 
this, was to make a conspicuous phone case disguised as 


a blood-glucose meter: 


P1: Putting a sensor on your phone, but you can also use it to 
see your readings, so you would unlock your phone and see 
the readings, have something that is made for one use, like 
this one the fingerprint detector, making it able to be used 
for another thing. 


This concept would allow the adolescent to use the 
medical device in public in a very conspicuous way. 
Combining the medical device with an object that 
used daily such as a cell phone, would also be more 


convenient. 


Both participants were interested in how technology 
could make their blood glucose meters smaller or 
concealed. They both thought of ideas that would 

allow for a blood-glucose sensor to be implanted into 
their veins. Participant 2 thought of an idea similar to a 
bracelet: "What if it was like a bracelet and the bracelet 
was in your vein? | don't know that would be painful 
though" and participant 1 had already seen tools that 
could be powered by blood and was intrigued by that 
prospect: "There are tools where it can be powered by 
blood, so if the sensor was a microchip it would be self- 
sustainable." These ideas show how the participants are 
interested in exploring the possibilities of technology 


creating smaller and more inconspicuous devices. 
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Both participants placed the images of the Hue inhaler 
(fig. 3) and Epipen (fig. 2) on their collage, as they both 
liked the colour options and that it did not immediately 
appear 'medical'. Participant 1 said that the Hue device 
offers personalisation “A lot of these are quite subtle. | 
think this one with all the colours, it doesn't make it look like 
its medical. It provides that personalisation." Participant 1 
also said "The EpiPen, that model will be good for an insulin 
pen. You look at it, and you don't think anything medical, it 
looks cool. It gives a different vibe." Both of these images 
show how personalisation and enhancing aesthetic 
appearance are strategies that could be explored in the 


future design process. 


Adolescent Preferences for New Concepts 

as Shown through Sketches. Both participants 
sketched the idea of an "all-in-one" device. They 

both found their current systems inconvenient, and 
participant 2 regarded carrying everything with them as 
annoying, "obviously there is not a time when you wouldn't 
need your insulin pen." Convenience is a theme that was 
mentioned repeatedly by both participants. Participant 
2's "all-in-one" device (fig. 6) is a portable testing and 
administering device. "You could have a three in one tool! 
So, it would be like your injection, finger pricker and blood 
glucose meter." This device would not only be convenient 
but also a more beautiful device that would not be 
recognised as easily by the public due to the casing 


surrounding the device. 
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Fig. 8. First concept drawn with participant 2 


Like participant 2, participant 1 wanted a convenient 
device and linked all his current devices together. 
They wanted the opportunity to create an all-in-one 
modular continuous blood glucose meter (fig. 7). The 
modular device would be worn on the arm, like the 
freestyle Libre. Participant 1: “Yeah, you could have lots 
of pieces! Ones got the insulin, ones got the battery, and 
ones got the Libre.” The participant liked the idea of 
technology developing far enough in the future where 
all their devices could be in one spot on the body and all 
linked to a phone app to control blood glucose levels, 
and insulin administration. This device combined two 
strategies: technological advancements and concealed 
the medical device from the public to create a more 


inconspicuous and convenient device. 


Participant 2 wanted to sketch a device that could be 
more “more innovative and stylish.” The participant was 
interested in materiality, texture, patterns and colours 
to create a more beautiful device. In the sketch (fig. 8), 
the participant wanted to engrave their name onto the 
insulin pen design. Naming the device demonstrates a 
sense of ownership over the concept and shows that 
they are proud of the idea they have generated. The 
participant even said they were proud of the work: 
“Wow, l've never done art, but | am kind of proud of 
this." The device they have designed could fit into the 
personalisation category but also the "enhance the 
aesthetic appearance of the traditional medical device" 


as a metal mesh encases it. The device could also fit 


into the strategy "disguise as an accepted non-medical 


device." 


4. Discussion. 


The participants were comfortable testing their 
blood-glucose levels and administering insulin in public. 
However, they both had experienced questioning from 
curious people at school and sports practices regarding 
their medical devices. The participants both saw these 
questions as harmless. However, there are occasions 
where participants are not in the mood to be asked 


questions. 


Adolescents had different preferences towards the 
five strategies developed and explored in this paper. 
Participant 1 was interested in technology, and how 

it could be used to make a more inconspicuous and 
convenient medical device and system. This was evident 
in the current devices the participant uses, which 
include a Freestyle Libre, insulin pump, MiaoMiao 
sensor and a smartwatch to receive constant readings. 
The participant did not want a device that appeared 
immediately "medical", and their preference towards 
inconspicuousness included wearing the device 
underneath clothing such as the Freestyle Libre. The 
semantics of using technology would allow their device 


to become smaller and more inconspicuous. 


Participant 2 had a different preference for their chosen 
strategy. They preferred a conspicuous device that is 
more "innovative and stylish." They were drawn to 
colours, patterns, textures and materials. This was seen 
in their sketches (Figure 6 & Figure 8), as they annotated 
specific materials and patterns that they would like 

to be seen in their concepts. Participant 2 was also 
interested in combining the personalisation strategy to 


allow for their device to be engraved with their name. 


Both participants had similar opinions about 
conspicuity. However, they recognised that this is 
not representational of all adolescents with type 1 
diabetes. Only two adolescents took part due to 
time constraints and access to participants. Although 
only two adolescents participated in this paper, the 
interviews were successful in understanding the 
participants' experiences of type 1 diabetes and 


provided a level of detail and story-telling that might not 


Design and Semantics of Form and Movement 183 


have been achieved through a questionnaire or survey. 
Participatory design methods also contributed to the 
detailed results. It allowed for the participants' opinions 
to contribute to the designing process, and the concept 


sketches are tailor-made to their needs. 


The next step is to explore further how the strategies 
can be applied to type 1 diabetes medical devices. 

The participants interviewed in this paper found the 
questions and activities engaging and will be returning 
for future feedback sessions with higher-fidelity 
prototypes. Further research in this area would be 
interesting to see how designers could operationalise 
the strategies. Additionally, medical devices may not be 
the only area where a conspicuous product is preferred, 
other areas of design such as safety products, assistive 
products and wearable technology might be other areas 
to explore how the semantic strategies could address 


the conspicuity of products. 


5. Conclusion 


In this study, five semantic strategies that address the 
conspicuity of medical devices were developed and 
explored with adolescents who have type 1 diabetes. 
Thematic analysis of literature that explores stigma in 
product design was used to develop the strategies. The 
strategies are: enhancing the traditional medical device 
to make it more beautiful, disguising the medical device 
as a non-medical item, concealing the medical device 
from the public, personalising the medical device for 
the user, and using technology to advance the medical 
device. These strategies all aim to design conspicuous or 
inconspicuous medical devices for adolescents with type 
1 diabetes. 


Two participants took part in semi-structured 
interviews, collaging, and sketching to explore the five 
semantic strategies. The semi-structured interviews 
successfully created an open discussion around testing 
in public, conspicuity and their preferences for their 
medical devices. The participatory design methodology 
allowed for a detailed conversation and provided more 
context and insight on the topic in addition to the 


sketching of potential ideas. 


Participants valued different strategies, in particular, 


using technology to advance the medical device and 
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to enhance the traditional medical device to make it 
more beautiful. These strategies were often combined 
with aspects of other strategies to produce their ideal 
concept for a conspicuous or inconspicuous medical 
device. Continuing this research will require further 
exploration of the semantic strategies, to determine the 
best combination of strategies for future concepts of 
medical devices. Designers will also play an important 
role in the next phase of the research to understand if 
the strategies can be used in other design disciplines and 


by practising designers. 


While the medical devices currently used by 
adolescents' do little to meet their preferences for 
conspicuity in public settings, this research has explored 
new strategies that could help to design new medical 
devices for adolescents with type 1 diabetes that can 


match their preferences for conspicuity. 
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Sitting Still: Seat Design for a New 
Head-Only MRI Scanner 


Abstract 


Magnetic resonance imaging is the gold standard in 
medical practice for tissue imaging. However, current 
devices are expensive and require users to be in a 
supine position during the scanning procedure, with 
users often reporting claustrophobia and anxiety. This 
study reports on the design of a patient handling system 
for a new kind of head-only MRI scanner, which allows 
users to be seated. This poses a unique challenge in 
seat design: the design should allow the user to be 

still to attain accurate MRI scans. We investigate the 
aesthetics of interaction in this complex dynamic system 
to yield a comfortable user experience. We followed 

a human-centred design process comprising two main 
design phases. An experimental setup was used to 
evaluate possible upright positions of the user. A refined 
functional prototype was consequently built and tested 
with healthy participants. Our results indicate that 

the position required to sit in a lounge chair is most 
comfortable and allows subjects to remain still for an 
extended time period. Our chair prototype provided 
sufficient support during the 20 minute procedure 

but needs further refinement to improve the user 


experience and usability of the system. 


Keywords 


MRI, medical design, seating design, design prototype, 


user experience 
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1. Introduction 


Magnetic resonance imaging (MRI) is a tremendously 
versatile technique for tissue imaging, with applications 
as diverse as cardiac health monitoring through 

to brain tumour detection. MRIs are used in five 

major anatomical regions: central nervous system, 
musculoskeletal, body, cardiovascular, and breast 
imaging [1]. To obtain accurate images of the human 
body, existing MRI scanners require the patient to 
remain motionless in a confined horizontal space. The 
enclosed space and extended time period that the 
patient is asked to remain still can cause enormous 
anxiety prior to and during the scan [2]. Techniques 
used in clinical practice that aim to reduce anxiety and 
motion during the scan include the provision of relevant 
information, playing music during the procedure [3]-[5], 
providing a mirror to look outside the bore [6] and 
adding foam pads [4], [7]. However, the process can 

be extremely challenging for patients. A feeling of lack 
of control [8] as well as lying on a stretcher during the 
procedure [4] have been identified as negative factors in 
the user experience and aesthetics of interaction. 

The development of a seated, head-only imaging 

system dedicated to brain imaging, which is one 

of the most common applications for MRIs, would 

offer a less restricting scanning procedure. It could 
furthermore contribute to a feeling of control due to 


the upright position and being able to look outside 


The patient is scheduled for an MRI scan 


INFORMATION 


The patient receives pre-examination advice 


the magnet using a small window. Such a novel system 
could be transportable, increasing access for a wider 
population, especially people living in isolated areas. The 
proposed new kind of MRI scanner is currently under 


development by a world-leading team of MRI experts 


[8]. 


We report in this study on the design of the seating 
user-experience from a human-centred design approach 
[9] and with consideration to the aesthetics of 


interaction [10]. 


2. Background 


Patients who are scanned with an MRI are required to 
lie on a stretcher, which is then moved inside the bore 
of the magnet. There are few MRI applications that 
allow a prone instead of the standard supine position 
such as the MRI breast scanner [11]. Further variations 
allow a seated position [12] or offer detachable and 
MRI-compatible seating elements to scan patients 

that cannot be imaged in the traditional position, or 
to gather specific imaging that requires gravity, for 
example, spine scans [13]. Extremity MRI systems [14] 
seem to provide a comfortable experience for the 
patient but do not allow a stable and upright position 
for the patient's head, which is one of the technical 


constraints that our design needs to address. 


PREPARATION SCAN 
PROCEDURE 


RESULTS 


The aesthetics of interaction refers to "the way 

an object speaks to us, calls us, affords us, puts us 

into contact with others, is meaningful to us" [10]. 
Aesthetics of interaction is experienced at the ongoing 
interaction space where the physical characteristics of 
the product and the information it conveys (functional, 
augmented, inherent) meet and converse back and 
forward with the user's context (general knowledge, 
aesthetic preferences, cognitive and affective states, 
personality, motivation) and sensory-motor system 
[15]. We reviewed design variables of MRI systems 

and other industrial chair designs, medical chairs and 
automotive car seat designs that would allow an upright 
position. Currently no other seating system exists that 
could be used as part of a head-only MRI scanner, let 
alone one that addresses the aesthetics of interaction. 
Interestingly, automotive car seat designs pay much 
more attention to the aesthetics of interaction than 
other reported medical device designs, by trying to 
create a beautiful experience as well as addressing 

the highly demanding technical constraints of car seat 


design. 


A number of studies focus on the experience of MRI 
systems and user experience of patients as well as 
technical staff [2], [4], [6], [8], [16] [20], but fail to 
mention explicitly how to design for the aesthetics of 
interaction. We developed a customer journey map 
based on these studies, see Figure 1 for an overview 
of the main milestones. The studies indicate that the 
MRI user experience is often extremely challenging, 
see Table 1 for details relating to the scanning process, 
which is the main focus of this study. It has been 
reported that up to 3776 of patients experience a 
moderate to high level of anticipatory anxiety [2]. The 
main contributor seems to be a feeling of claustrophobia 
[17] and associated negative emotions already start to 
set in before the actual scanning procedure takes place 
[19]. The anxiety can even manifest itself after the scan 
affecting the daily routine of the patients [17]. It has 
been pointed out that anxiety and claustrophobia are 
strongly influenced by a feeling of suffocation, ideas of 
control and the potential for harm to be caused by the 
machine [8]. Our new design system aims to address 


the aesthetics of interaction by providing elements 


Fig. 1. Milestones of the user experience. We focus in this study on the that evoke a feeling of control to improve the MRI user 


orange part of the experience. experience and remind the user of similar systems 
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they are familiar with. In particular, we took car seat 
design as an area to draw inspiration from for comfort, 


familiarity and aesthetics of interaction. 


3. Aim 


The aim of this study was to suggest an optimal upright 
seating position for a head-only MRI scanner, which 
contributes to an improved MRI user experience while 
securing minimal motion of the user in combination with 


a head restraint system we will describe in later work. 
4. Study Context 


This study (funded by the National Institutes of Health 
(USA) and Kiwinet (INZ)) is part of a multi-institutional 
research project taking place in the United States, Brazil 
and New Zealand. The novel head-only MRI system is 
currently under development and could therefore not 
be used as part of our testing protocol. However, we 
used a physical non-functional representation of the 
scanner bore as part of our user testing sessions. We 
report in this paper specifically on the design of the 


seating for the new MRI system. The study was carried 


out by a team of industrial designers with experience in 
developing medical technology and a technical expert in 


the development of MRI magnets. 


Constraints defined by engineering and clinical factors 
included: to design comfortable seating; head should be 
aligned to the magnet bore and its movement restricted; 
magnet cannot be moved when positioning the patient; 
maximum width and height of the seating area; materials 
need to be hospital graded if fabric and nonmagnetic if 
structural. This differs from traditional seating as users 


are not free to move their heads. 


5. Materials and Methods 


In the context of developing new medical devices, an 
iterative design procedure assessed by a number of 
usability methods is an ideal setup as part of a human- 
centred design approach despite being expensive and 
time-consuming [9]. We followed a human-centred 
design approach [21] with two main design phases: 


1) an initial design exploration phase to investigate 


possible positions of the user in a sit-up MRI scanner Table 1. Customer 


journey milestones 


Step Positive factors Neutral factors Negative factors 

Start of 1. music used as a form of distraction 1. lack of control over the situation 
scanning 2. entering the bore feet first can 2. not liking or not having any music at all 
procedure decrease anxiety 


The patient has . possibility to have a nap 


to remain still . noise cancelling headphones 
. mental distraction 

. support from others 

. rewards and/or bribes 


. music 


breath holding techniques 


ON Oo Uo d € Mo 


constant reassurance that the 
patient is in control of the situation 
9. sedation 

10. eye masks 

11. physical touch by the MRI 
technician or family member 


12. buzzer that allows to contact staff 


13. trust in staff 


1. not all patients want to 
actively participate in the 
scanning process 

2. there can be a preference 
to hand over responsibility 


to expert 


1. amount of time spent in the system 
2. inability to keep still 

3. pain 

4. claustrophobia 

5. feeling unable to breathe properly 
6. panic symptoms 


7. threat to self-control 


9. desire to scratch an itch 


10. thoughts about 
11. having panic symptoms 
12. lying on a hard bed 


13. temperature is too hot or too cold 


machine. 
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8. enforced nature of having to remain motionless 


he (negative) consequences 


14. thoughts about the potential harm caused by the MRI 


through observation, idea generation and low fidelity 
prototyping; and 2) a refined fully functional prototype 
evaluated by healthy participants. 


5.1 Design Development Process 


In a first step, we used an experimental design process 
to investigate patient positions in the novel MRI scanner 
that would allow an upright and stable position of the 
head for up to 30 minutes. A low fidelity prototype 

[22] was used to investigate where support needs to 

be provided in a seated position, within the specific 
constraints provided by the engineering team. We 

used pillows, a raised horizontal platform, cardboard 
boxes and polyurethane foam sheets to investigate how 
support may facilitate motionless yet comfortable sitting 


positions. 


In a second step, we used expansive design concepts 
[21] exploring different designs and positions of the 
chair in the overall system based on the initial results. 
We followed a systematic approach whereby we 
assessed the design concepts against a list of design 
criteria [23] developed for this project and experience 
prototyping [24] to refine the design concepts until the 
prototype had reached a level where it could be tested 
with participants. A full scale of the chair was produced 
using a foam core and fibreglass laminate to provide the 
required stiffness and robustness. Adjustable upholstery 
was provided to the participants using a selection of 
different foam thicknesses. Similarly, elements for the 
lumbar region and movable inserts for the hip area were 


used in the assessment process. 


5.2 Evaluation 


A combination of different evaluation methods is helpful 
to determine a range of usability issues [25]. We chose 
a multi-method approach in the form of an ergonomic 
analysis [26] and formative usability testing [26] with 
participants. Even though the role of good design is 
recognised in the context of developing medical devices, 
it seems that ergonomic principles are often under- 
used [9]. We employed standard data sets of human 
dimensions [27] and results of a study on overweight 
populations [28] for the ergonomic analysis process. An 
additional qualitative scale of comfort and discomfort, 
which was adapted from Helander and Zhang's chair 


comfort scale [29], was used as part of our testing 


protocol with healthy participants to confirm the 

results of the ergonomic analysis process. The aim 

of our formative evaluation process with a full-scale 
prototype was not the collection of quantitative data for 
comparison, but instead to refine the design [30]. We 
took the participants through a simulation of an MRI 
examination and asked them to remain as motionless as 
possible in the MRI bore during the 20 minute testing. 
The testing included the sound of an MRI scanner, 

arm movements to evaluate the degree of movement 
tolerable and semi-structured questions about comfort, 
discomfort and feelings of anxiety in regards to the chair 
and overall system to address aesthetics of interaction. 
Participants could leave the system at any stage. We 
used the system usability scale (SUS) [31], and an MRI- 
Anxiety Questionnaire [32], which measures anxiety 
and relaxation during the MRI examination as part of 
the evaluation process. We used a 10 points Likert scale 
as part of the questionnaire. The testing was video and 


audio recorded for later analysis. 


5.3 Recruitment 


This study was approved by the human ethics 
committee of Victoria University of Wellington (VUW). 
Participants provided informed written consent 

before taking part in the testing. We recruited healthy 
participants through VUW. Ideal sample size suggestions 
for usability testings vary between 3 to 20 participants 
with 5-10 participants being a sensible approach for 
most studies [33]. Four participants took part in our 
first design experiments and seven in our formative 


usability testing. 


5.4 Analysis 


The photos, videos, notes of observations and semi- 
structured interviews were analysed using thematic 
analysis in a deductive form [34]. Thematic analysis 

is used to identify, report, and analyse patterns and 
themes within the data. Themes were generated by the 
first author and reviewed by the remaining authors. The 
initial themes were translated into design criteria [23] 
as per Table 2 below. The design criteria were used at 
different stages of the research and design process to 
systematically assess the design concepts and refine 
them. The results of the scales and the questionnaire 


were analysed using descriptive statistics [35]. 
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Criteria description Based on 

1 Chair fits below the MRI Physical project constraints 
scanner 

2 Width and height fits the | Literature: Human factors 
5th to 95th percentile 

3 Positions the user MRI usability constraints 
correctly in the middle of 
the bore 

4 Minimises movement of RI usability constraints 
the user 

5 MRI safe materials and RI usability constraints 
medical appearance 

6 Reduces feelings of Literature: MRI user 
control and anxiety experience 

7 Allows arm movement of | Literature: MRI user 
the user experience 

8 Easy entering and leaving | Physical project constraints 
of the magnet more 


Table 2. Assessment of design concepts against design criteria 


6. Results 


6.1 Design Development Process 


Experimental Design. Four participants took part 
in the initial phase of our experimental setup involving 
the use of low fidelity prototypes to test possible 
comfortable positions that ensure a straight and stable 
position of the head. Two participants were female and 
two were male. Their age range was between 22-23 
years. Participants indicated that a position required to 


sit on a chair is the most comfortable for an extended 


Patient remote 
control 


Technician remote 
control 
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period of time in the given physical constraints of this 
research. The participants mentioned that support is 
required for the lower back and the regions around the 
lumbar, neck, under the legs and buttocks to ensure 
comfort. Armrests and support around the thighs were 
identified as unnecessary. Participants acknowledged 

a requirement of softening of the seating-pan edge to 
reduce discomfort. All participants acknowledged a 
need for leg support however results were inconclusive 


as to the degree of support needed. 


Expansive Design Concepts. For phase 2, we 
developed a range of concepts based on the results of 
our low fidelity prototyping. The concepts were then 
assessed against the updated design criteria detailed in 
Table 2. 


Full-size Prototype. The findings suggest a position 
similar to a car seat or lounge chair may be appropriate 
to facilitate head movement restraint. This gave us the 
opportunity to address the aesthetics of interaction 
similarly to (and expand on) the car seat industry. 

The final chair design consisted of a seating pan and a 
separate, moveable backrest, see Figure 2. The chair 
was positioned on an electric lift column that allowed 
the user to be positioned correctly inside the magnet 
bore and could be controlled by the participant or by 

a research assistant. This approach was chosen based 
on the feeling of lack of control that multiple studies 
indicated. The chair prototype was tested with a 
physical representation of the final magnet bore and 
frame. We used an air ventilation system that ended 
just below the bore entrance, which could be regulated 
in terms of its intensity to provide sufficient air flow. 
For reference, the seat and upholstery are coloured 


black to assist in motion tracking of in-scan participant 


Fig. 2. Fully functional 


Sene Bre DE RUP test rig with removable 


lumbar support and 
Air ventilation system . 
upholstery inserts. 


Adjustable chair backrest 


Chair seating pan 


Electric lift 


Table 3. SUS value: 
0 = low and 


100 =high,n=7 


Fig. 3. MRI Anxiety 

questionnaire with a 

10 points Likert scale 
(n=7) 


movement, which we are planning to carry out in 

the second stage of this research to confirm that our 
system minimises head movement to under 1mm. 
Seven participants took part in the testing. Two were 
female and five were male. Participants were between 
20-58 years and had a height between 168 -190 cm. 
Their weight ranged between 50-90 kg. 


System Usability and User Experience. The results 
of the SUS indicated a high degree of overall usability 
of the system, see Table 3. However, a few participants 
expressed a need for more relevant information on 
how to use the system and a low level of confidence 

in using the system. The results of the MRI-Anxiety 
Questionnaire [32] was used to evaluate factors of 
anxiety due to the enclosed space and prolonged time 
that participants had to remain still. The results suggest 
that participants felt mainly relaxed, however, some 
participants expressed concerns regarding the ability 
to breath; control over the situation; and the feeling of 


being safe, see Figure 3. 


Physical Usability Issues. We informed participants 


before the testing that the armrests were not 


completely weight bearing, which caused most to use 
the raised edges of the seating pan to lower themselves 
down into the chair. Participants had no issues leaving 
the chair system, however, some participants noted that 
some of the provided larger upholstery inserts created 


difficulties in entering the system. 


The feet, neck and shoulder region at the back were 
pointed out as body parts that most participants felt 

a moderate to high level of discomfort with. Most 
body parts started to feel uncomfortable over the 
extended period of the testing session when the 
participant had to remain motionless inside the magnet 
bore. In particular the footrest was requested to 
decrease physical stress on the ankles and feet while 
the participants were elevated off the ground. Further 


notable areas for improvement were: 


1. The angle of the backrest should be reclined; 

2. The length of the seating pan needed extension; 

3. The chair seemed to be positioned too high when at 
the lower travel limit of the lifting column (50cm); and 
4. The armrests should be designed in a way so they 


‘hugs’ the user. 


Participant | 01 02 03 04 05 06 07 
SUS value 70 95 73.5 90 42:5 85 72.5 
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£ m Participant 2 
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7. Discussion 


This study reports on the development and evaluation 
of a chair for a sit-up, transportable MRI scanner, with 
the very specific requirement of keeping the user's head 
still for a period of time. This study is based on two 
design phases which informed the design of a full-scale 
prototype that has been tested by participants. 

Our formative usability evaluation indicates that the 
chair provides sufficient support and comfort for a 
20-minute scanning process, but requires further 
refinement to improve the user experience, level of 


comfort and usability of the system. 


The SUS indicates a high level of usability of our 
proposed chair design. However, there were a number 
of observed usability issues that require further 
refinement. The entering and leaving of the system 
requires the user to bend their head in order to sit 
down. Arm rests were observed as an important feature 
that allowed a secure lowering of the user into the 
seating pan. The participants could use a remote control 
to position themselves inside the magnet bore, however, 
just a small number of the participants chose to do so. 
This might explain why some participants mentioned a 
lack of control over the situation, which is an important 
aspect in this design related to the aesthetics of 
interaction. The controller was difficult to use while 
being inside the scanner due to a lack of tactile feedback 
of the button that made it impossible to know which 
button needed to be pressed to move up or down. The 
participants were provided with a variety of upholstery 
elements to increase comfort. Some of the thicker 
upholstery parts compressed over the testing period 
consequently lowering the user further down, which 
meant that they either started adjusting themselves 
during the testing or could not look outside the 

scanner window anymore. The airflow was described 

as sufficient by some participants but not all of them. It 
was furthermore noted that the air felt very cold due 

to the high intensity. A different position of the air hose 


could potentially direct the air further inside the bore. 


The assessment of the aesthetics of interaction 
produced mixed results. Car seat design was a starting 
point to relate users to a familiar object they would 
know how to interact with; this was successful in part 


due to the anthropometry and posture required for this 


192 Design and Semantics of Form and Movement 


project, similar to that of a car seat. However, the travel 
necessary for the chair to bring the patient into the 
magnet, our requirement to fit 9576 percentile of the 
population, and the fact that the controls to move the 
chair are in a handheld remote control, meant that the 


aesthetics of interaction were different to a car seat. 


Some of the chair features were experienced to become 
uncomfortable over the 20-minute testing sessions such 
as the 90 degree position of the seating pan in relation 
to the backrest that required the user to sit upright 
evoking lower back pain in some of the participants. 

A recent study focusing on the development of an 
extremity MRI scanner [14] suggests a reclined position 
for the user. The study suggests a leg rest angle between 
130-148 degrees which we are going to take into 
consideration for the next iteration of the chair. Most 
of the participants recommended adding a leg rest and a 
foot support to avoid their feet dangling in the air during 
the examination. The lack of footrest evoked a high level 
of discomfort during our testing sessions in nearly all 

of the participants. This is an interesting finding as the 
first application of reclining chairs was for sick patients 
and for pregnant women to find a comfortable position 
without having to lie down (34). This form of chair has 
become associated with a level of comfort allowing the 
user to elevate his or her feet to increase comfort (34). 
Our study indicates that if the user of a chair cannot 
touch the ground, a leg and a footrest are essential 


feature to increase the user experience. 


Limitations include that the initial testing had a small (« 
10) sample size, with healthy and mostly young (« 30) 
participants. The mock-up of the MRI scanner simulated 


user experience in a lab environment. 


Based on the feedback of the participants we are 
currently changing the design of the chair for a second 
phase of usability testing. Further studies are required 
to confirm the usability of such a chair in a clinical 
environment, where patients and MRI operators 

are interacting with the system. However, given the 
high SUS scores of our current chair solution, we 
believe that following a final design iteration we will 
significantly improve subject experience during brain 
MRI examinations. This will have been achieved through 
consideration of the aesthetics of interaction in the 


critical chair design feature of our novel MRI system. 
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Designing Research Prototype for the 


Elderly: a Case Study 


Abstract 


This paper describes a research study regarding 
intergenerational story sharing of the elderly living in 
the nursing home, including four iterations, applying 

a Research-through-Design approach. It started from 
an exploration prototype named Interactive Gallery 

(1st iteration), and its findings helped to narrow down 
our research area and define our research question. 
To answer it, the prototype named Slots-story (2nd 
iteration) and Slots-memento (3rd iteration) were 
designed and implemented, which focused on life story 
and memento story of the elderly respectively. While 
the 4th iteration aimed at facilitating intergenerational 
story sharing and sustainably. The above research 
iterations offer an example of how research prototypes 
supports to focus research area, and answer the 
research question in stages. We finally conclude with 

a discussion of insights on designing prototype for the 


non-tech-savvy elderly. 


Keywords 


Research-Through-Design, Research Prototype, Elderly, 
Storytelling, Tangible Interface 


1. Introduction 


The aging society is coming. The worldwide population 
over age 65 is expected to more than double from 357 
million in 1990 to 761 million in 2025, and with up to 
50% of those over the age of 85 likely to be placed ina 
nursing home at some point in their lives [1]. However, 
social isolation is widespread among older adults 

in nursing homes, and older residents have limited 
involvement in social connections [2]. Living separately 
with their children also makes it difficult for younger 
and older family members to communicate with each 
other because of different interests, stereotypes 

of aging, geographical distance, and the fast pace of 
contemporary life. Since social interaction of the elderly 
is a broad research area, our research focuses on its 
subset: intergenerational storytelling of the elderly living 


in nursing homes. 


Given that one of the most precious characteristics 
of older adults is their memory of events, people and 
places [3], intergenerational storytelling could act as 
an effective way to keep them stay in touch with their 
children. Our target group is the aged non-tech-savvy 
people living in a nursing home. While younger seniors 
are embracing online social technologies, the Internet 
and social media use drop off significantly for people 
age 75 and older— Only 34% of people in the G.I. 
Generation (born in 1936 or earlier) use the Internet, 
and 21% have home broadband [4]. 


Design and Semantics of Form and Movement — 195 


In this paper, we describe our three-year research 
study of intergenerational story sharing of the elderly, 
in a Research-through-Design manner. It offers an 
example of how research prototypes support us to 
focus research area, answer research question in stages. 
We then present our insights on designing research 


prototype for the non-tech-savvy older adults. 


2. Related Work 


Social technology for the elderly. Existing research 
indicates that their social interaction could be promoted 
either by strengthening connections between older 
adults and their existing social circles (friends, family, 
etc.), or expanding their social circle by knowing more 
friends [5]. Since TUI (Tangible user interface) has been 
identified as having great potential to improve older 
adults' acceptance of technology acceptance [6], related 
applications for non-tech-savvy older adults mostly 


adopt tangible interface. 


Research-through-Design. R-t-D is described in the 
literature as an approach for scientific inquiry, taking 
advantage of the unique insights gained through design 
practice [7]. Although the term of R-t-D is not a new 
concept, it is until recent years that it has been widely 
discussed and used in the HCI field, and became an 
increasingly recognized approach in design. R-t-D is 
conceptualizing research done using the skillful practice 
of design activity, revealing research insights [8]. One of 
the features lies in that it acknowledges and embraces 


professional practices’ contributions to knowledge[9]. 


Research Prototype. R-t-D also highlights the 


importance of research prototype. The use of 


designedly prototype within the research process 

has been well acknowledged. Prototypes are defined 

as research instruments created by the researchers, 
tailored to each individual study [10]. Prototypes 

serve multiple functions within the research process: 
Prototypes make abstract theory concrete as they could 
involve people in research process, and they are 'like 
products' in the sense that someone can interact with 
them and experience them [11]. Prototypes are also 
seen as embodying designers’ judgments about valid 


ways to address the possibilities. 


Summary. Our research focuses on the storytelling 
between older residents in the nursing home and their 
children. The older adults are story producers, while 
their children are the memory trigger producers. 
R-t-D and research prototype have been widely 
discussed and used in HCI field, but there are few 
research that specifically discuss the application study 
of R-t-D for the elderly people. In the next section, we 


explain how our R-t-D cycles evolve in detail. 


3. Iterative Design Cycles 


The research question of our project is: How can we 
use design to facilitate intergenerational storytelling and 
preservation for older adults living in the nursing home? 

To answer this question, three sub-questions are 
formulated: (1) What are their stories about? (2) In which 
ways, can design facilitate them to tell and preserve stories? 
(3) In which ways, can design involve their children? The 
entire research project, including the above research 
question, was done in a Research-through-Design 


manner. 
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3.1 First Prototype: Focus Our Research 
(Define RQ) 


As social interaction of the elderly is a broad and 
macroscopic research area, our first prototype 
Interactive Gallery was an explorative prototype, 
designed to uncover the related potential factors, 
narrow down research goal and define the research 
question. Details process could be seen in [12], and the 


following is a short description. 


Contextual inquiry. We firstly conducted semi- 
structured interviews with seven older adults and two 
caregivers in a Dutch nursing home. We found older 
adults encountered difficulties in connecting with their 
fellow residents. The nursing home was not a fully open 
community, which made it a relatively isolated and 
independent. Most of them couldn't operate digital 


devices, and they still highly relied on physical operation. 


Prototype. In response to this, we designed the 
Interactive Gallery system (Figure 1-(1)), which consisted 
of a set of scenery-collectors and a gallery-like 
interactive installation. The formers are distributed to 
volunteers from local communities to share real-time 
scenery photos with older adults, and the latter is 
placed in the nursing home which enables older adults 
to watch and start conversations. Communication 
between sharers and receivers will also be connected 
through a “postcard-sending” metaphor: The older 
adults could print scenery photos as postcards by simply 
pressing the button. They could choose to send back to 


sharers, or keep them. 


Field Study. It was implemented in a Dutch nursing 
home for five weeks. Direct observation method 
(including baseline observation and intervention 
observation) were conducted, and semi-structured 
interviews were contacted semi-structured interviews 


with 13 older adults (ranged in age from 71 to 86). 


Finding. The field study proved the tangible interface 
employing metaphor reduces using barriers for them. 
Their memories were evoked by the familiar scenery 
photos. They preferred to share memories with their 
children, rather than the unacquainted volunteers. 

The above finding inspired us that the elderly could be 
deemed as content (memory, story, etc.) producers, and 


their children were the story listeners. This drove us to 


narrow down our research goal and further identify our 


research question. 


3.2 Second lteration: the Elderly's Life Stories 
(Answer rq1,rq2) 


The following is a brief description, and a detailed 
process could be seen in [13]. We firstly conducted 
semi-structured interviews with both the older 

adults and their children, and the following design 
requirements were defined: Memory trigger. Tangible 
interface employing metaphor. Using audio as the 
storage medium of stories. Could be used either 
face-to-face or separately by older adults and their 
children. Then three concepts were built based on 

the design requirements and were developed to be 
mock-ups(Figure1-(2)). Older adults were consulted 

to evaluate them. Slots-story employing metaphor of 
slots-machine was chosen and further detailed. Slots- 
story, a slots machine-like device, aims to facilitate 
inter-generational life story sharing and preservation. 

It utilizes with the metaphor of slots machine, and 
integrates functions of memory cue generator, story 
recording, and preservation. By default, there are 40 
trigger questions covering most aspects of an entire 
life. It could either be used face-to-face or separately by 
older adults and their children. In the field study, eight 
pairs of participants from a Dutch nursing home were 
recruited to use the prototype for around ten days, and 
each pair consisted of an old adult (six female and three 
male, ranged in age from 77 to 89) and his/her child. 
Semi-structured interviews were conducted with both 
older adults and the young. Stories were transcribed 


and analyzed. 


Reflection: As some stories were related to their 
mementos, such as album, souvenir, etc., which were 
also ideal memory triggers as they provided visual 
clues. This inspired us to explore their mementos and 
related stories in the next iteration. Additionally, the 
appearance of porotype should be refined according to 
older adults' feedback. 


3.3 Third Iteration: Their Memento Stories 
(Answer rq1,rq2) 


Research Prototype. We interviewed older and 
young adults separately. We first asked them to 


arrange a brief guided tour of their homes, aiming to 
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examine their mementos for displaying and stored 

in hidden places. The prototype was refined based 

on the interview and feedback of the 2nd iteration: 
Prototype's decorative effects needed to be highlighted 
to make it unobtrusive when putting it at older adults' 
home (Figure1-(3)). It was used in a cross-generational 
operation manner: the young took photos of older 
adults’ mementos and copied them to the prototype, 
while older adults used the prototype to tell stories 


related to the mementos [14]. 


Field Study. Ten pairs of participants (each pair 
consisted of an old adult and child) were recruited 

to use it for around seven days, and mementos and 
recordings were collected. Semi-structured interviews 
were conducted with older adults and their children. 
Mementos were categorized and analyzed. Stories were 


firstly transcribed, then were analyzed. 


Reflection. Sustainability of the intergenerational 
sharing was necessary. Next, a cellphone application will 


be designed for the young side. 


3.4 Fourth Iteration: Involve Their Children 


(Answer rq3) 


Aim. In this iteration, we close the intergenerational 
storytelling loop by designing an App for the young 
generation, aiming to facilitate intergenerational story 


sharing and preservation in a sustainable manner. 


Prototype. A system consisting of a slot machine-like 


device used by older adults, a cellphone application used 


by the young. It integrates life story and memento story 
sharing (Figure 2)[15]. 


4. Discussion 


We have illustrated our R-t-D cycles including four 
research prototypes, which provide insights on how 
R-t-D is applied to research practice, especially 

for the elderly people. In this section, we present 

the implications in two parts: Prototype as tools of 
identifying and answering research question, and 
designing research prototypes for the non-tech-savvy 


elderly. 


4.1 Prototype as Tools for Identifying and 
Answering Research Question 


Prototype Acts as a Means of Narrowing Down 
Research Focus. First, Design research is a way to ask 
larger questions beyond the limited scope of a design 
problem, through the practice of design itself. 

Our research focus and detailed research question 
were identified through the first prototype Interactive 
Gallery. As mentioned, social interaction of the 

elderly is a broad and macroscopic research area: 
their social connections with fellow residents, with 
family members, old friends, people from the local 
community, caregivers, .etc. Interactive Gallery was 
firstly implemented, and its findings helped us to focus 
our research on intergenerational storytelling of older 


adults 


(Section 3.1). Next, new and unexpected findings 
emerged during the implementation of prototype, 
which guided our follow-up research. Our second 
prototype was regarding life stories of the elderly. 
However, new and unexpected findings emerged during 


its implementation: some stories were related to family 
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Fig. 2. Prototype of 
4th iteration 


Fig. 3. Interview 


with older adults in 


different iterations 


mementos, such as album, souvenir, artwork, etc. This 
inspired us to explore their mementos and related 


stories in the next iteration (Section 3.3). 


Answering Research Question in Stages through 
the Evolution of Prototypes. Our overall research 
question was answered in stages. RtD itself has 

highly iterative character as the design evolves while 
conducting RtD-led research [16]. As mentioned, we 
break the overall research question into three sub- 
questions. To answer rqf and rq2, we designed and 
implemented the 2nd and the 3rd prototype, which 
focused on older adults' life stories and memento 
stories respectively. To answer rq3, we designed the 
4th prototype, to fully involve the older adults' children. 
The evolution of prototypes is not only a response to 
the development of research questions and insights, 
but also the feedback of the participants. For example, 
the appearance of the 3rd porotype was refined based 
on the feedback in 2nd iteration, as the elderly thought 
it should be unobtrusive when putting it at home, 
therefore decorative and vintage effects needed to be 
highlighted. 


During the research process, we switched between 
reflection and practice. Knowledge is gained by 
conducting a design exercise and continuously reflecting 
on direct and indirect observations, beliefs and 
experiences[16]. In our R-t-D cycles, reflection is the 
output of the implementation of the current prototype, 
also the input for the next prototype. Therefore, 
reflection is the dividing line between analyzing and 
practice: it is the end of a phase of study, but the 
beginning of a new phase of study. The reflection is a 


catalyst for knowledge generation. 


4.2 Designing Research Prototypes for the Non- 
Tech-Savvy Elderly 


Non-tech-savvy Older Adults. One thing to note 
is that, since people above the age of 65 years old are 
diverse regarding cognitive ability, they compromise a 
group that is considerably more diverse than people 
of the general (younger) population. As such, the 
experience of using technology of each individual 
older person is unique, and their level of technological 
mastery varies. Therefore, they could be roughly 
divided into the non-tech-savvy and tech-savvy group. 
Our target group is older adults in nursing home, 
according to literature and our investigation, most of 


them are non-tech-savvy users. 


The following are some insights on designing prototypes 
for the non-tech-savvy elderly derived from our 
research practices. Since all of our prototypes adopted 
tangible interface, the presentation of our insights 

is partly based on Eva Hornecker et al's framework 

on tangible interaction. The framework is structured 
around four themes: Tangible Manipulation, Spatial 
Interaction, Embodied Facilitation, and Expressive 
Representation. Each theme concludes serval 


concepts[17]. 


Providing Haptic Direct Manipulation through a 
Tangible Interface. 

It has been well acknowledged that tangible interfaces 
are more accessible and suitable for the needs of 
elderly people, as they could provide a natural style of 
interaction [1]. The elderly are suffered from declines 
in motor control and accuracy, while physical contact 
with an interface gives elderly users confidence in their 
abilities[18]. Current touch-based interfaces are mainly 
visually guided without physical feedbacks. This lack 

of tangible feedback often leads to several errors and 
frustrations that are accentuated in older people. Haptic 
direct manipulation refers to tactile contact, haptic 


feedback, and material qualities when manipulating the 
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interaction objects[17]. All our research porotypes 
adopted tangible interfaces providing haptic direct 
manipulation. Interactive Gallery was equipped with big 
buttons, and the elderly could manipulate it by a simple 
hand-press. Slots-story and Slots-Memento provided 
tangible interfaces and intuitive interactions through 
the lever. Both operations were accepted by the elderly 
users according to our field study. As one older adult 
said in our interview: “ Its function is simple. | especially 


like the lever, and the knob is easy to hold." 


Providing Representational Significance through 
Metaphor. Representational significance refers to the 
interrelation of physical and digital representations, 

and how users perceive them[17]. It is important to 
communicate information with analogies and metaphors 
that are relatable to the systems that older adults are 
familiar with, and a new product that is designed in 
familiar metaphors can reduce the barriers of elderly 
users to use [14]. Metaphors of the gallery, postcard- 
sending, and slots-machine were applied our prototypes 
respectively. The interaction styles were based on the 
elderly's familiar knowledge to help them understand 
easily. During our interview of the first iteration, one 
older adult said: "It's like a photo art gallery, and it is 
better because the photos are dynamic." For our second 
iteration, one older adult said: "The slots-machine-like 
operation raises me a sense of expecting and curiosity for 


the unknown." 


Providing Paper Instructional Manual. As our 
prototypes were used by the elderly independently, 

the instructional manual was needed to tell them 

how to operate the prototypes. During our interview 
of the first iteration, one older adult said: “It is too 

novel for me, | dare not use it." Another said: "Actually 

| dare not to press the button and | ask others or my son 

to print postcard for me." Research also indicated the 
elderly have reported that they are afraid of using new 
technological devices as they are afraid of damage the 
device [19]. Therefore, in the 2nd iteration, we provided 
the elderly participants with paper instructional 
manuals, which consisted of step-by-step instructions 
and corresponding illustrations (Figure 4). The paper 
manual could also afford easy annotation, which was the 
elderly were generally more familiar with, compared 
with online content. During our study, we found older 


adults in the nursing home still relied on newspaper 
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to get information. The instructions in paper manuals 
are static, which are easier for some older users, as it 
matched their learning style. Research indicates that 
older adults have a stronger preference for using the 
device's instruction manual over trial-and-error because 


it matches their learning style. 


Tailoring Interface to the Elderly: Aesthetic and 
Visibility. User interfaces need to be tailored to the 
specific user group—the elderly. Two aspects were 
considered in our case, aesthetic and visibility. 

Firstly, according to our study, elderly people were still 
interested in traditional physical objects, vintage and 
old-fashioned were in line with the aesthetic view of 
the elderly. Therefore, the vintage style was applied in 
the appearance of the Interactive Gallery. While in the 
interview with the elderly regarding the appearance 

of Slots-Story, its decorative and vintage effects needed 
to be highlighted as they hoped prototype could be 
unobtrusive when putting it at home. As one older adult 
said during the interview of the 2nd iteration: “I think 

I have enough house appliances in my home, | don't want 
another one." “I don't use digital devices, and | think the 
vintage style could bring a sense of mysteriousness." “The 
vintage style could give indications of past things." Secondly, 
big and bold fonts need to be adopted considering fading 
the eyesight of the elderly. The capability of the eye 

to focus on near objects is diminishing for the elderly. 


Fig. 4. The paper 


The visual presentation of information should consist of — . 
instructional manual 
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the older adults were suffering from a decreased vision: 
"Two things that have great influences on my life, bad 
mobility and bad eyesight." 


Avoid Accurate Operation. Large movements rather 
than delicate operations were applied in our prototypes. 
The elderly are suffering from the decline in fine 

motor skills and accuracy of movements, which makes 

it harder to use small buttons and switches. In our 

case, big button and lever were designed in Interactive 
Gallery and Slots-story respectively, to compensate for 
the elderly suffering from the decline in the accuracy 

of movements. As one older adult said during our 
interview: “I used to listen to radios, and | still have one in 
my room. But | feel difficult to operate precisely when | scroll 
through stations by turning a knob." Some older adults 
mentioned the sensitivity of the Slots-story's operation 


should be reduced, as their hands were clumsy. 


Providing Embodied Constraints through 
Minimization of the Number of Interface 
Components. Embodied constraints refer to the 
physical set-up constraining users' behaviors. This 
restriction eliminates the possibility of making errors, 
and eliminating the possibility of errors could lower 
the anxiety brings to the elderly. Our experiences 

of providing Embodied constraints is minimizing the 
number of interface components: In Interactive Gallery, 
there was only one interface component--a big button. 
While in Slots-story, there were only two operation 
components: a handle and a group of buttons. During 
our interview, older adults appreciated the simple 


interaction. 


5 Conclusion 


In this paper, we report our research study regarding 
intergenerational story sharing of the non-tech-savvy 
elderly people living in nursing homes, applying R-t-D. 
We first discuss the research prototype as tools 

of identifying and answering the research question, 
including prototype acting as means of narrowing down 
research focus, answering research question in stages 
through the evolution of prototypes, and switching 
between reflection and practice. These lessons have 

a universal significance for R-t-D application study. 
We then present our lessons of designing research 


prototypes for the non-tech-savvy elderly, including 


providing Haptic direct manipulation through tangible 
interface, providing Representational significance 
through metaphor, providing paper instructional manual, 
tailoring interface to the elderly: aesthetic and visibility, 
avoid accurate operation, and providing Embodied 
constraints through minimization the number of 


interface components. 


References 


1. Reichman, W. E., Coyne, A. C., Borson, S., Negron, A. 
E., Rovner, B. W., Pelchat, R. J., ... Hamer, R. M. (1998). 
Psychiatric consultation in the nursing home: a survey of six 
states. The American Journal of Geriatric Psychiatry, 6(4), 
320-327. 
2. Mor, V., Branco, K., Fleishman, J., Hawes, C., Phillips, C., 
Morris, J., & Fries, B. (1995). The Structure of Social 


Engagement among Nursing Home Residents. The Journals 
of Gerontology Series B: Psychological Sciences and Social 
Sciences, 50B(1), P1—P8. doi:10.1093/geronb/50B.1.P1 

3. Dryjanska, L. (2015). A social psychological approach to 
cultural heritage: memories of the elderly inhabitants of 
Rome. Journal of Heritage Tourism, 10(1), 38—56. doi:10.108 
0/1743873X.2014.940960 

4. Madden, M., Lenhart, A., Cortesi, S., & Gasser, U. (2010). Pew 
Internet and American life project. Washington, DC: Pew 
Research Center. 

5. Waycott, J., Vetere, F., Pedell, S., Kulik, L., Ozanne, E., Gruner, 
A., & Downs, J. (2013). Older adults as digital content 
producers. In Proceedings of the SIGCHI Conference on 
Human Factors in Computing Systems (pp. 39—48). ACM. 
Retrieved from http://dl.acm.org/citation.cfm?id=2470662 

6. Spreicer, W. (2011). Tangible interfaces as a chance for higher 
technology acceptance by the elderly. In Proceedings of the 
12th International Conference on Computer Systems and 
Technologies (pp. 311—316). ACM. 

7. Frayling, C. (1993). Research in art and design. 

8. Zimmerman, J., Stolterman, E., & Forlizzi, J. (2010). An 


analysis and critique of Research through Design: towards a 
formalization of a research approach. In Proceedings of the 
8th ACM Conference on Designing Interactive Systems (pp. 
310-319). ACM. 

9. Godin, D., & Zahedi, M. (2014). Aspects of research through 
design: a literature review. Proceedings of DRS, 1667—1680. 

10. Boer, L., Donovan, J., & Buur, J. (2013). Challenging industry 
conceptions with provotypes. CoDesign, 9(2), 73-89. 

11. Soegaard, M., & Dam, R. F. (2012). The encyclopedia of 


Design and Semantics of Form and Movement 201 


12. 


13; 


14. 


15. 


16. 


17. 


18. 


19: 


202 


human-computer interaction. The Encyclopedia of Human- 
Computer Interaction. 
Li, C., Lin, X., Kang, K., Hu, J., Hengeveld, B., Hummels, 

C., & Rauterberg, M. (2018). Interactive Gallery: Enhance 


Social Interaction for Elders by Story Sharing. In M. 
loannides, J. Martins, R. Zarnié, & V. Lim (Eds.), Advances 
in Digital Cultural Heritage (pp. 104—116). Cham: Springer 
International Publishing. 

Li, C., Hu, J., Hengeveld, B., & Hummels, C. (2018). Slots- 
story: facilitate inter-generational life story sharing and 
preservation of the elderly. In Proceedings of the 10th 
Nordic Conference on Human-Computer Interaction (pp. 
691—695). ACM. 

Li, C., Hu, J., Hengeveld, B., & Hummels, C. (2019). 

Slots- Memento: Facilitating Intergenerational Memento 


Storytelling and Preservation for the Elderly. In Proceedings 


of the Thirteenth International Conference on Tangible, 
Embedded, and Embodied Interaction (pp. 359—366). New 
York, NY, USA: ACM. doi:10.1145/3294109.3300979 

Li, C. (2019). Designing a System to Facilitate 
Intergenerational Story Sharing and Preservation for Older 
Adults. In Proceedings of the Thirteenth International 
Conference on Tangible, Embedded, and Embodied 
Interaction (pp. 737-740). ACM. 

Reeker, L. F., & Brazier, F. M. T. (n.d.). Lessons Learned from 
Research through Design, 7. 

Hornecker, E., & Buur, J. (2006). Getting a grip on tangible 
interaction: a framework on physical space and social 


interaction. In Proceedings of the SIGCHI conference on 


uman Factors in computing systems (pp. 437—446). ACM. 
Retrieved from http://dl.acm.org/citation.cfm?id=1124838 
áikió, J., Wallin, A., Isomursu, M., Ailisto, H., Matinmikko, T., 
& Huomo, T. (2007). Touch-based user interface for elderly 


users. In Proceedings of the 9th international conference 


on Human computer interaction with mobile devices and 
services (pp. 289—296). ACM. Retrieved from http://dl.acm. 
org/citation.cfm?id=1378021 

Laguna, K., & Babcock, R. L. (1997). Computer anxiety in 
young and older adults: Implications for human-computer 
interactions in older populations. Computers in human 


behavior, 13(3), 317-326. 


Design and Semantics of Form and Movement 


Topic 7. 


Designing with Humans, 
Machine Intelligence and Data 


Design and Semantics of Form and Movement 203 


Giuseppe Bono!, Pilar Maria Guerrieri? 
‘UCL - The Bartlett School of Architecture, ?Politecnico di Milano 
ucbqono@ucl.ac.uk 


pilarguerrieriQ)polimi.it 


PLUG-INS JUNGLE 
Algorithmic Design as 


Inbuilt Dynamism 


Between Human and Artificial Creativity 


Abstract 


The paper seeks to define the concept of plug-ins jungle 
as generative background in computational design for 
new forms of dynamic relationship between human and 
artificial creativity. This definition will be conducted 
through the use of theoretical references and practical 
case studies with the intent to show that the current 
evolution of a plug-ins jungle is affecting design results 
at all its scales — from product design to urban design 

— and at all its stages — from design conception to 
construction and fabrication. 

In the past twenty years, the creation of software 
applications during the design process has exponentially 
increased in every design-related disciple and the use 
of plug-ins is giving birth to a new algorithmic evolution 
of creativity. One of the most interesting examples 
regarding such matter corresponds to the graphical 
algorithmic editor Grasshopper and the generation 

of a series of animal-named plug-ins through which 
graphically editing the coding process. Grasshopper and 
its plug-ins represent the touchstone of the ongoing 
evolution of a new digital jungle in which new software 
applications are not only augmenting human creativity, 
but they are also generating new forms of artificial 
creativity in themselves. 

The evidence suggests that the increasing use and 
creation of software applications during the design 


process is giving birth to new forms of dynamic 
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relationship between human and artificial intelligence, 
an inbuilt dynamism through which the human mind 
and the machine constantly interact with each other 


producing unexpected and emergent design results. 


Keywords 


Plug-Ins, Algorithmic Design, Human Creativity, 
Artificial Creativity, Design Ethic 


1. Introduction 


In 2014, Andrew A. Smith publishes his book 
Grasshopper Jungle [1]. Despite being a young adult 
fiction and coming-of-age novel quite unusual and 
original for its literary genre, its title sounds particularly 
familiar if referred to algorithmic design and robotic 
fabrication. In fact, the graphical algorithmic editor 
Grasshopper currently represents one of the most 
popular and developed algorithmic editors for designers 
available on the market. Its use is widely spread 
throughout the majority of design-related disciplines 
which are now developing new algorithmic techniques 
to explore new ideas and opportunities during the 
design process. The rise of such algorithmic approach is 
transversally affecting the design process at all its scales, 
breaking down boundaries between disciplines which 


were far from each other during the past years. As a 


matter of fact, from urban design to product design, 
algorithms are providing new platforms of research 
contributing to the generation of emergent and 
autonomous design results. 

The popularity of an algorithmic editor such as 
Grasshopper lies on a series of opportunities which it 
is able to provide to its users. First of all, Grasshopper 
allows the creation of graphical-based algorithms 
generated by an inductive process which allows the 
user to intuitively connect pre-coded nodes. Secondly, 
it allows the user to personally code new scripts 
through the use of several programming languages 
such as C£, Python, and VB. Finally, the generation of 
new and totally customized plug-ins is then promoted 
inside a unique platform in which the same plug-ins can 
interact with each other. The inbuilt connection with 

a 3D modeling software such as Rhinoceros ultimately 
provides the user with the opportunity to physically 
produce the designed object through several production 
techniques, such as the creation of STL files for 3D 


printing or specific scripts to program robotic arms. 


The innovation which this approach is bringing into the 
design process is clear and significant, and it is giving 
birth to new forms of relationship between human 

and artificial creativity. These new forms of evolution 
require an analysis able to consider algorithms not only 
as mere mathematical tools, but rather as examples of 
the rise of a new human progress in design conception 
based on the influence of artificial intelligence over 


human mind and perception. 


Nowadays, the interaction between human and 

artificial intelligence represents an open field of debate 
and investigation, and such ongoing discussion about 
people in favor of or against to one side or the other 
has been well depicted by Max Tegmark in his book 

Life 3.0. Being Human in the Age of Artificial Intelligence 

[2]. In such a background of interest and research, a 
series of questions arise: what is the nature of this new 
relationship between human and artificial intelligence? 
How is it affecting the design process and the new forms 


of creativity generated from it? 


The answer to these two questions requires the use 
of two concepts, one theoretical and one practical. 


The former is related to the theoretical point of 


view to adopt in evaluating the relationship between 
human and artificial creativity, while the latter refers 
to the practical ways through which such relationship 
is manifesting itself. The first concept is the concept 
of inbuilt dynamism between human and artificial 
intelligence, while the second one corresponds to the 
idea of plug-ins jungle. Since the former represents the 
theoretical point of view through which looking at the 
latter, a note on the inbuilt dynamism between human 
and artificial creativity is required before explaining the 
factors which are influencing the rise of the current 


plug-ins jungle. 


2. Note on the Inbuilt Dynamism Between 
Human and Artificial Creativity 


The expression 'inbuilt dynamism' refers to the 

analysis done by the cognitive scientist Margaret 

Boden in regards to the work of Harold Cohen and 

his Al system AARON, one of the first example of 
computer-generated art developed by the British artist 
during the 1970s. Commenting on Cohen's work and 
developing the same theoretical construction in her 
book The Creative Mind. Myths and Mechanisms [3], 
Boden highlights the fact that computational processes — 
including scripts, frames and semantic nets — are helpful 
to understand how the brain works and how some 
aspects of human creativity are possible. The reason for 
it is "because symbolic and representational structures 
and transformations are the focus of computer 
programming, the essence of creativity may not be so 
far removed from computational processes as is usually 
assumed" [4]. Although the existance of similitudes 
between the human brain and computer was something 
already well presented in the studies of computational 
pioneers such as Alan Turing and John Von Neuman 

[5], Boden provides an interesting interpretative key 
through which reading such complementary relation 


between artificial and human intelligence: 


A functioning program has its own inbuilt dynamism. Its 
activities can be both flexible and constrained, and a proper 
amalgam of flexibility and constraint is central to creative 


intelligence. [6] 
Then, flexibility and constraint as the two main factors 
to understand the real measure of complementarity 


between artificial and human intelligence, a balance 
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which can only be fully appreciated through the 


awareness of their inbuilt dynamism. 


Another important contribution to the idea of dynamic 
complementarity between human and artificial creativity 
is provided by Kostas Terzidis. In his book Algorithmic 
Architecture Terzidis focuses the attention on the new 
computational process in architecture describing 

the use of algorithms not only as mere step-by-step 
problem-solving procedures, but rather as ontological 
constructions with philosophical and sociological 
repercussions. Starting from clarifying the distinction 
between computation and computerization — something 
already well explained by the author in his previous 
publication talking about 'algorithmic form' [7] — 
Terzidis coined the term ‘algotecture’ to highlight the 
use of algorithms in architecture. Proceeding through 
the explanation of a brief history of such 'algotecture' 
as something substantially different from the more 
common Computer-Aided Design system — the former 
not necessarily dependent on the computer, while the 
latter dependent by definition on it — Terzidis gives 
evidence to the fact that there are certain levels of 
problems which cannot be solved by standard CAD 
systems, but their complexity inevitably required the 
use of algorithms. His words are particularly significant 


in this regard: 


There are some problems whose complexity, level of 
uncertainty, ambiguity, or range of possible solutions 
required a synergic relationship between the human mind 
and a computer system. Such synergy is possible only 
through the use of algorithmic strategies that ensure a 
complementary and dialectic relationship between the 


human mind and the machine. [8] 


Such ‘complementary and dialectic relationship between 
the human mind and the machine' highlighted by Terzidis 
explains the role of algorithmic design in comparison 
with human creativity through a critical point of view 
which can be reasonably reconducted to Boden's 
concept of inbuilt dynamism in functioning programs and 
their creative intelligence. This critical point of view is 
based on mutual persuasion and compromises which are 
able to generate new forms of creative dynamism inbuilt 


in the new artificial dimension of human creativity. 
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The analysis of these theoretical references has 

been conducted with the purpose to clarify the 
complementary relationship between human and 
artificial creativity. Such point of view represents the 
foundation upon which looking at algorithmic design 
and evaluating the results coming from it. The ongoing 
phenomenon of mathematization of the design process 
is based on the creation of software applications and 
this approach is characterizing the most advanced 
researches in contemporary design. New plug-ins are 
created and grouped together inside bigger platforms 
— such as Grasshopper — and sometimes they are even 
programmed to constitute stand-alone applications. 
Both cases represent the evidence of an exponentially 
increasing 'plug-ins background' which is progressively 
evolving the traditional conception of human design: this 
wide and open landscape is represented by the plug-in 


jungle. 


3. Plug-ins Jungle 


The expression plug-ins jungle refers to AD Profile 

222 Computation Works. The Building of Algorithmic 
Thought guest-edited by Brady Peters and Xavier De 
Kestelier in 2013. Inside the issue, a series of examples 
are presented to analyze the effects of computational 
design inside the architectural discipline. In the article 
Design Eco-System. Customizing the Architectural Design 
Environment with Software Plug-ins, Daniel Davis and 
Brady Peters describe the evolution of architectural 
design towards the use of scripting and personalized 


plug-ins: 


This defines an entirely new landscape in which ‘cathedrals’ 
(monolithic applications) are challenged by ‘bazaars’ 
(generative-modeling editors) populated by animal-named 
plug-ins. [9] 


In the following pages, a series of Grasshopper plug- 
ins are described such as Kangaroo, Pachyderm 
Acoustic Simulation, Weaverbird, Geco, Firefly. If 
now we add to them other Grasshopper plug-ins — 
such as Goat or Bowerbird, just to mention a couple 
— and even compound software such as Rhinoceros 
and its rendering extensions — Penguin or Flamingo 
among the others — the definition of a plug-ins jungle 
seems pertinent and appropriate for a computational 


environment dominated by animal-named applications. 


Inside the same AD Profile 222, a series of case 
studies are proposed with the purpose to highlight 

the effects of such algorithmic tools over the final 
design results. In this regard, particularly interested 
are the examples explained by Giulio Piacentino and 
the use of the Grasshopper plug-in WeaverBird [10], 
and Thomas Grabner and Ursula Frick and the use of 
another Grasshopper plug-in such as GECO [11]. In the 
first case, the use of the plug-in gives access to surface 
subdivisions and transformation operators which allow 
a topological editing of the starting design creation. In 
doing so, the exploration of a series of design options 
comes from the action performed by the algorithm 
itself, which therefore becomes a fundamental factor 
in design creation. In the second case, the use of 

the plug-in GECO in conjunction with the software 
Autodesk Ecotect allows a constant interaction 
between modeling and software analysis, and such 
interaction produces effects in terms of design results 
— such as in the case study analyzed in the article, that 
is the Shenzhen Boarder Station competition entry 
designed by SPAM, where the plug-in has been used to 
run a solar access analysis through which roof openings 
have been located and designed. In both cases, the 

use of plug-ins significantly influences the final design 
results, and therefore they are clear examples of how 
the action of the plug-ins jungle is a touchstone of the 
rising phenomenon of hybridization between human 
and artificial intelligence in the current evolution of 


creativity in computational design. 


The examples which can be mentioned to confirm 
such fundamental influence of plug-ins over final 

design results are numerous and almost unlimited. In 
fact, nowadays the use of Grasshopper and its plug- 
ins is becoming part of the standard practice in many 
architectural firms, and it is very often the case to see 
such applications embedded into the workflow from 
the very early stages of the design process constituting 
basic tools of design exploration and investigation. 
Having said that, the concept of plug-ins jungle goes far 
beyond being a definition related to a single algorithmic 
editor such as Grasshopper, but rather it represents 

a more complex working model towards which both 
the design profession and the construction industry 
are evolving too. Algorithmic creations and software 
applications constitute by now the core of any advance 


research related to the built environment, and this 


aspect represents a clear confirmation of the fact that 
the ongoing process of mathematization is not only 
producing new forms of development inside the single 
discipline, but more importantly it is linking together 


different disciplines inside new algorithmic platforms. 


4. Extended Examples of Plug-Ins Jungle 


In the current evolution of computational design, there 
are several fields of research in which the action of the 
plug-ins jungle plays a central role in the realization of 


final design results. 


First of all, plug-ins jungle is the natural habitat for the 
complexity of design generated in the last twenty 

years across all design-related disciplines. Promoted 

by the rise of a mass-customized production system, 
sophisticated software applications and new digital 
fabrication technologies have enables designers to build 
experimental structures with a high level of complexity. 
Such complexity is based on continuous and seamless 
forms, and although nowadays such trend seems to 

be overtaken by the rise of a more discrete approach 

— in this regards, Lei Zheng offers an interesting 
comparison between the two models [12] — the 

digital turn of design in the last two decades has been 
based on a constant exploration of complex shapes 

and structures. For instance, the work conducted at 
ZHACODE is particularly significant in this regard. The 
computation and design group at Zaha Hadid Architects 
is a research group which focuses its attention on 
invention and innovation through the use of computer 
software programming and physical computing, 
considering algorithms as exploratory phases for the 
discovery of new design and production opportunities. 
Among a wide range of projects, the installation 
‘Thallus’ exhibited in Milan during Salone del Mobile 
2017 represents a perfect example of the research 
conducted by ZHACODE in terms of customization and 
mechanization of the design process. The installation 

is composed by a 7km long continuous line made of an 
extruded structural strip produced by 6-axis robotic 3D 
printing technology. The design of the structural strip 

is based on differential growth methods dictated by 

an algorithm which uses specific parameters — such as 
proximity to boundaries, angled direction of ruling, and 
structural requirements — to establish density gradation 


and direction of growth. 
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Other than representing the field of action for complex 
forms and structures in digital mass-customized 
production, the plug-ins jungle is also the place for 

the creation of tools through which promoting open- 
source design. The rise of open-ended and adaptable 
systems based on sharing software, ideas and models 

is becoming one of the main field of research in the 
current evolution of computational design throughout 
all the disciplines. An interesting example of this 
approach is represented by the work done by Enriqueta 
Llabres and Eduardo Rico in terms of urban analysis 
and design. Through their multidisciplinary London- 
based office Relational Urbanism, Llabres and Rico 
developed a design approach to digital forms of urban 
documentation based on Relational Urban Models 
(RUM s). As explained by the authors in their article 
Relational Urban Models. Parameters, Values and Tacit 
Forms of Algorithms [13], RMUs are design models based 
on the new forms of digital urban documents which 
collect inputs from designers, government bodies and 
members of the public, allowing information sharing 
and feedback from the end user to the design team 

and vice versa. This new form of shared authorship in 
design conception is possible through an application 
based on a generative algorithm able to analyses data 
and parameters to elaborate new urban configurations. 
For instance, parameters based on the proximity of 
the street network allow to retained certain blocks 
and to remove certain others, or again the final 
configuration of a tower can be sculpted on top by 
constraints of sunlight exposure. The final design is the 
result of the action of algorithms able to mediate the 
effects of environmental parameters with the starting 


requirements coming from human inputs. 


Another field of research in which the plug-ins jungle 
plays an important role is automation and digital 
production. For example, the wide use of 6-axis robotic 
arms in design production and assembly is made possible 
by the use of algorithms and software applications 
which not only allow to program robots according to 
specific design requirements, but also influence the final 
design results through the control of the robot's action 
and movement. For instance, the Grasshopper plug-in 
Robots is a clear example of this process and it allows 
programming ABB, KUKA and UR robots. Always 
talking about robotic fabrication and the influence 


on design played by algorithms to program such 
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machines, an interesting example is provided by the 
heterogeneous multi-robot systems which constitute an 
open field of research at the ICD/ITKE at the University 
in Stuttgart. As explained by Maria Yablonina and Achim 
Menges [14], such multi-robot systems are based on 
the use of bespoke design machines in conjunction 

with standard industrial robots. This mix during the 
fabrication process allows a higher level of flexibility and 
scalability compared to the exclusive use of industrial 
robotic arms. An example of this process is provided 

by the ICD/ITKE Research Pavilion 2016-17. In this 

case, the use of a path correction algorithm relying on 
camera-based tracking system controls the flight of the 
mobile robot which is represented by an unmanned 
aerial vehicle (UAV), while other software applications 
are used for analysis and optimization of the Pavilion 
structure dictating robots’ movement through the 
location of the primary and secondary structures, both 
made of carbon tensile filament structures with the 


primary one realized in carbon fiber-reinforced ribs. 


Finally, another field of research in which the action of 
the plug-ins jungle plays an important role in the creation 
of final design results is represented by optimization 
and form-finding techniques. As already explained by 
Mark Burry describing the works of Antoni Gaudí and 
Frei Otto as main precursors in computational design 
in terms of form-finding and structural optimization 
[15], nowadays such techniques are widely used in 
contemporary design conception and they are based on 
several software applications. For instance, Kangaroo 

is one of the most popular plug-ins for Grasshopper 
and its use allows to modify design in response to 
engineering analyses simulating aspects of the behaviour 
of real-world materials and objects. Always regarding 
such matter, the work of the Digital Structures 
research group at the MIT Massachusetts Institute of 
Technology represents an important example in terms 
of structural optimization and the different design 
configuration generated from it. Group leader Caitlin 
Mueller pays particular attention to the relationship 
between structural optimization and design conception 
in her article Distributed Structures: Digital Tools for 
Collective Design [16]. Highlighting the fact that the 
creation of new computational tools is shifting the 

role of computation itself from representation and 
analysis to creative idea generation, Mueller provides a 


series of examples in which the use of multi-objective 


optimization techniques offers the opportunity for 

the designer to choose between different options 
generating during the optimization process. Such 
opportunities are made possible to the use of plug-ins 
specifically developed for such purpose. For instance, 
the web-based design application StructureFIT and 

the Rhino and Grasshopper plug-in Stormcloud — both 
developed by the MIT Digital Structures research group 
— allow to designers to explore new design typologies 


and forms with high level of structural feasibility. 


The examples described so far represent only four 
instances belonging to a wide range of case studies 

in which the final design is the result of a very close 
relationship between the human input and the 
computational calculation belonging to the machine. 
From this relationship, a new design world is rising and 
the plug-ins jungle represents its natural habitat. Such 
habitat is based on the constant interaction between 
the human mind and the machine which effects design 
right at the very beginning of its conception. Inside such 
contextual background, plug-ins are the places able to 
host new forms of emergent creativity coming from a 


new relationship between the man and the machine. 


5. Conclusion 


The theoretical considerations and practical case studies 
analyzed so far represent a critical point of view through 
which looking at the current evolution of computational 
design and the increasing use of software applications 
during the design process. The rise of the plug-ins jungle 
is affecting computational design at all its scales — from 
product to urban design — and at all its stages — from 
design conception to automated fabrication. Software 
applications are becoming indispensable prerequisite for 
any design process and the fact that the authorial role of 
the human designer may not survive to this digital turn 
is a concrete possibility for the future of any design- 
related profession. As already explained by Mario Carpo 
in his book The Alphabet and the Algorithm, the more 
frequenwt use of algorithms may eventually transform 
human designers into IT developers, i.e., changing their 
role from creators to mere plug-ins users/generators 
[17]. This scenario may not represent an alarming 

point for those ones who believe that the future will 

not belong to human beings but to machines. On the 


other hand, this point of view may not be supported 


from those ones who believe that any technological 
development is first and foremost a matter of human 


progress. 


The theoretical construction of the entire paper is built 
upon the association between two apparently dissimilar 
species — such as the man and the machine — and the 
inbuilt dynamism which regulates such relationship. 
Considering the design process as a field of investigation 
has constituted an appropriate point of view to 
understand the dynamic complementarity between such 
species and the creativity which arises from them. After 
all, as already explained by Nicholas Negroponte at the 


very beginning of the computational evolution of design: 


The partnership is not one of master (smart, leader) 
and slave (dumb, follower), but rather of two associates 
which each have the potential for self-improvement 
[...]. Eventually, a separation of the parts could not 
occur; the entire ‘symbiotic’ system would be, as Gordon 
Pask described, an artificial intelligence that cannot be 
partitioned. [18] 


According to Negroponte, from the rise of this new 
form of artificial intelligence, a new 'extended designer' 
(the man) and 'artificial designer' (the machine) will 

be generated in favor of a constant and mutual design 
complementation, augmentation, and interaction 
between the human mind and the machine. Today, 

such 'extended designer' and 'artificial designer' are 
two actors of the same reality and their interaction is 
constantly producing new forms of inbuilt dynamism 
between human and artificial intelligence. The 
complementary approach between them seems to be 

a more appropriate point of view from which looking 
at the ongoing evolution of algorithmic design, and a 
closer look into its natural habitat, namely a walk in the 
plug-ins jungle, may provide an effective starting point for 


further research. 
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Defining a Data Impact Tool for 


Design Courses 


Abstract 


Wearable product/service solutions are increasingly 
spreading in everyday life aiming at supporting and 
tracking activities, also increasing self-awareness about 
behaviors and body parameters. The amount and variety 
of personal data generated and collected by these digital 
services is unprecedented in human history, and impacts 
on individuals, societies and organizations can now be 
only partly predicted. The design of data management 

in these solutions has so become a critical task for both 
designers and service providers. This paper presents 
the results of the application of a design tool, as part 

of an Impact Anticipation Method, created to foresee 
problems and opportunities so to improve design 
choices in quick design processes and design courses. 
The tool aims at stimulating the discussion on impacts 
related to the use of personal information. The results 
point out the usefulness of the method’s tool in raising 
awareness in designers, elicit critical thinking fostering 
the discussion on the topic, and provide insights for 
improvements. It demonstrates the need of more 

agile ways to address the topic in the design process. 
Students faced their lack of knowledge about how and 
when personal information is implied in their solutions, 
and on potential impacts that such information could 


have. 


Keywords 


Personal Information, Design Tool, Impact Anticipation 
Method, Ethical And Societal Impacts 


1. Introduction 


Wearable product/service solutions are increasingly 
spreading in everyday life aiming at supporting and 
tracking activities mainly for wellbeing and healthy 
purposes, also increasing self-awareness about 
behaviors and body parameters. Data-driven solutions 
commonly employ personal data to provide value 
within services, and the management of such data 

has become a critical task for designers and service 
providers. The design of a new product/service is often 
carried on with the purpose of responding to needs 

of people and communities; in other words, design 

for innovation is often directed toward utopian goals. 
However, innovative solutions and functionalities, when 
implemented in the society, can lead to unexpected 
consequences related to some elements that compose 
the solution, or to the use of the solution itself. As 
pointed out by visionary technologists and scholars 

[1] [2] [3] [4], the use of personal data in services 

has a social relevance as an agency that could have 
impacts on the way people perceive themselves [5][6] 


[7], on how the perceive the contexts they interact 
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with, and on how they participate in social life [8][9] 
[10]. It also changes actions and behaviors [5][6][11], 
as well as roles and relationships between people and 
between people and organizations [12][13][14][15]. 
Ackoff's DIKW model [16] defined data as “symbols 
that represent the properties of objects and events and 
their environment". Data "are useless until they are in 
a useable (i.e. relevant) form" [16]. Taking persons as 
carriers of properties, we consider personal data as 
raw material to be processed so to be useful. Personal 
information is then the signifier extracted from the 
elaboration of personal data. While the authors of this 
paper are engaged in an extensive research about the 
use of personal data in design [17][18], in this paper we 
focus on personal data involved in wearable devices. 
Wearable technologies and sensors embedded in 
wearables gather data about individuals and process 
them so to create personal information that is valuable 
for the users and for service providers. Personal 
information is interpreted by services through analysis 
and comparison with rules, reference knowledge, 
statistics and algorithms; in this way, it becomes 
knowledge that can be used to provide added value 
through service functions or provide awareness to the 
user itself. The amount and variety of personal data 
that are generated and collected by digital services is 
unprecedented in human history, and the impacts on 
individuals, societies and organizations that can now be 
only partly predicted. From the design point of view, 
the management of data is therefore a new dimension 
of social responsibility in design, requiring suitable 
attention and efforts. In other words, as designers, we 
argue the importance of developing design approaches 
apt to exploit the potentials of data to produce value 
for individuals and communities, and, on the other 
hand, to learn how to predict the possible — utopian 
and dystopian - impacts of functionalities and services 
employing data, so to adopt suitable design choices. 
The main research developed an approach (knowledge, 
method and tools) for the anticipatory investigation of 
the potential impacts of the management of personal 
data in product/service design [19][20]. In this paper 
we focus on the use of a light (reduced and adapted) 
approach, to be adopted in education and whenever 
the design resources do not allow the use of our main 
methodology. The paper reports the results obtained 
applying the light approach in an education workshop, 


212 Design and Semantics of Form and Movement 


and the main features of the proposed approach. The 
results demonstrate that the anticipation activities 
provide design hints and guidelines for the responsible 


development of services and systems. 


2. Exploring Impacts Related to Personal 
Data in the Design of Wearable Products/ 
Services 


To foresee the impacts that an interactive service could 
have when implemented in the society, we defined the 
Impact Anticipation Method (IAM). The method brings 
the society's point of view into the design process to 
encourage and support ethical discussion on future 
impacts of solutions that imply the collection and use 
of personal data and personal information. It identifies 
insights to raise awareness and stimulate critical thinking 
during the design phases enabling the identification of 
the potential impacts. Using this approach, for each 
specific project brief, we can identify critical themes 
related to the use of personal information that we 
consider as relevant for the assessment of features of a 
new project [19][20]. 


For the specific topic of this paper, we report here 

the critical themes that can significantly be related to 
the design of wearable devices involving the use of 
personal data. Each theme should be considered during 
the design process so to orient the creation of a new 
solution toward the optimal satisfaction of user needs 


and to the maximum respect of basic individual right. 


FORESEE IMPACTS 


DESIGN-CHOICES SOLUTION 


DESIGN PROCESS @ 


USES IN THE SOCIETY 


OPPORTUNITIES PROBLEMS 


Fig. 1. The IAM aims to foresee issues (opportunities and 


problems) during the design process 


Awareness of Data Tracking, Sharing and Use. 
The users of a wearable connected object are supposed 
to be well aware of detection and tracking of behaviors 
and biological data in real-time. What if they are not 
fully knowledgeable? Some issues arise when users are 
not completely conscious about when the tracking 
occurs, which data is collected, with whom every kind 
of data is shared or sold, who is using it and what kind 


of profit can be obtained by providers and partners. 


Altered Cognitive Load. Proactive and reactive 
services can lower cognitive load allowing the user 

to shift the focus from tasks to the user experience. 
However, especially in the case of wearables, the use 
of the technology itself can increase the cognitive load 
due to information and knowledge returned as visual 
feedback or insights. Furthermore, wearables can raise 
stress due to information and suggestions related to 
problems, options and situations the user wasn't aware 


of. 


Self-mirroring into Data. The users' self-perception 
changes and relates to the self-knowledge they acquire 
while understanding their own's data [21]. When the 
users access the data, their ability to conveniently 
interpret the returned information is crucial with 
respect to the improvement of self-knowledge and 

to avoid misinterpretation. Services should therefore 
provide the information consistently with the user's 


interpretation skills. 


Altered Attitude and Quality of Life. Tailoring and 
automation can improve already existing services and, 
as well, enable the creation of new ones for specific 
purposes. The increasing availability and pervasive 

use of sensors can change users' attitude in actions 

and behaviors. People behave in a different way when 
they know (or think) they're being observed [22]. A 
system can however take advantages from the 'observer 
effect' aiming at changing users' behavior for their good 


according to goal settings. 


Creation and Management of Community of 
Value. The interaction of people through their data 
often creates or reinforce a community thanks to the 
sharing of values related to the purpose of data tracking. 
As an instance, a community of runners can sort out 


clusters based on running performance. Crucial tasks 


are to ensure that the exposition of data among the 
community is volunteer and to consider, in the design 
process, possible issues related to self-exposure and 
self-disclosure; as well, the lack of representation in the 
community of people that don't track themselves should 


be considered. 


2.1 The Impact Anticipation Method 


The Impact Anticipation Method is the main outcome 
of a PhD research [23] and it eventually aims to elicit 
critical thinking in designers so to support decision- 
making during the design process of innovative solution. 
The raising of awareness of possible impacts, allows 
designers to create more robust and reliable products 
strengthening also the image of the company that is 
providing the service. The method is a twofold system: 
the first one consists in the Potential Issues Database 
aimed at the creation of knowledge about the 
possible consequences, - problems and opportunities 

- that the use of personal information in connected 
services might produce. The second part of IAM 
consists in the application of this knowledge in 

the design process through the Data Impact Tools so to 
raise awareness in the designers about possible impacts. 
The Potential Issues Databases is a rich collection of 
references, reporting situations associated to the use 
of personal data in services and connected products, 
coming from real world discussions and from future 
scenarios extracted from fiction. The references 

are classified and therefore explorable through an 
interactive platform [24]. The knowledge included in 
the Potential Issues Database ranges between different 
application fields and can be employed in different 
design contexts. Following the IAM, the impacts related 
to the use of personal data and information can be 
expressed in four layers: i) user's self-perception; ii) 
people's behavior and performance of actions; iii) 
interpersonal relationships; iv) social agency [19][20]. 
The iteration of the tools' design has been performed 
applying and refining them in use cases of research 
projects and design courses. This paper focuses on the 
second phase of the method and reports results of a 
research investigating the potential of the approach 
suitable in design contexts with string constraints 

of time and resources. Possible applications of the 
knowledge contained in the Potential Issues Database 


refer to four main activities of the design process: 
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Analytic. In the preliminary analytic phase, designers 
explore the context for which they have to design the 
interactive solution. In this phase, the Potential Issues 
Database provides suitable refences through research 
filters (Impact Layers) of use cases and situations, so to 
foster the discussion on user's rights and produce design 


guidelines for the subsequent design phases. 


Creative. Designers ideate the concept through the 
envisioning of context scenarios of application. In this 
design phase, the Potential Issues Database supports the 
envisioning of scenarios through the analysis of insights 
coming from eight macro-scenarios that we created 

to represent in a synthetic way the most important 
situations that can be associated to data management 
[18]. 


Refinement. Designers deal with specific choices 
(physical design) and formalize the solution through 
prototyping and refinements. The knowledge contained 
in the database support the decision-making about 
functionalities. The application of the Data Impact Tool 


in this phase is the specific focus of this paper. 


Assessment. The assessment phase allows designers 
to evaluate the solution and test it with users through 
prototypes. The Potential Issues Database supports 
the evaluation of features and the preparation and 
conduction of user tests so to center the attention 
on ethical and social elements to create guidelines for 


future iterations of the project. 


3. A Light Approach to the Investigation 
of Impacts Related to the Use of Personal 
Information 


Design tools are the means by which designers preside 
over the cornerstones of the process, so to ensure 
that all the most critical and important aspects are 
nevertheless carried forward and considered. Designers 
use visualization to develop design thinking, to share 
knowledge with project partners, to create consensus 
with stakeholders, and to guide the development and 
implementation of the material parts and those not 
tangible of a system [25][26]. In other words, design 
tools are a designer way to produce knowledge and 

to support the suitable development of projects. The 


IAM approach is not just focused on the production of 
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Fig. 2. Application of the knowledge of the Potential Issues 


Database in the design process 


knowledge, but our research also includes experiments 
to investigate how the knowledge can be applied 
through design tools in different contexts. For these 
reasons, we have developed a light tool to be used in 
design education and in the design-contexts with limited 


resources. 


3.1 The Creation of the Light Data Impact Tool 


In previous work we described some applications of 

the method [19] and its iteration [20] that led to the 
creation of an In-depth Data Impact Tool meant to be 
used by expert designers in complex and long design 
processes. However, the validation of the In-depth Data 
Impact Tool highlighted limits related to its complexity 
and time needed for its use. We created the Light Data 
Impact Tool that is meant for the design contexts where 
the full data-base is not available or when the time 

and resources for design are limited, consists in some 


design activities, specifically focused on the investigating 


Fig. 3. Students working on on-body sensations and related 


body map of stimuli used as reference for the creation of the 


concept 
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the potentials and criticalities connected to the use of 
personal data for a specific design brief. The features of 
the light tool are explained in the following of the paper 
through the report of its application in an education 


experiment. 


3.2 The Workshop About On-Body Interaction 
Design 


The workshop "In Close Proximity" was conducted by 
Professor Patrizia Marti and Laura Varisco at Politecnico 
di Milano in June 2018 with a class of students (around 
40) of the master's degree program in Digital and 
Interaction Design as a reflection on the experience 

of use of on-body technologies [27]. It guided the 
students through a research-through-design exploration 
following a step-by-step process to design wearable 
solutions, from the initial idea to the interaction 
dynamics using various design tools (e.g. Body Map, 
Mood board, Personas, Bodystorming, Wizard of Oz, 


interviews, and questionnaires). 


The students dedicated one full day of the two-week 
workshop to the exploration of possible impacts of the 
use of personal information of their designed solutions. 
We used this workshop to validate an agile Light 

Data Impact Tool that can be use in education and by 
professional designers. The tool has been used during 
the definition phase, when designers already had a clear 
concept for their solution and were defining features 
and details. The application of the tool followed four 


steps: 


Step 1 - Lecture. A lecture to clarify the topic of the 
use of personal information, and the related types of 


opportunities and problems. 
Step 2 - Identification of Implied Personal 


Information. A template was distributed, and 
students had 30 minutes to fill it with: A written 


Session 1 (Teams 2,4,6,8) | Session 2 (Teams 1,3,5,7) 


(1h) Template 2 fulfillment 


(1h) Group conversation 


(1h) Template 2 fulfillment 


(1h) Group conversation 


Table 1. Organization of the activities for the two sessions 


description of how and why their solution implies 
personal information; The identification of all the actors 
involved in the service and their motivation; The list 

of technologies implied to gather and manage personal 


information. 


Step 3 - Exploration of the Potential Issues. In 
order to have comparable results, we divided the 8 
teams of students in two sessions. The first session 
spent one-hour in a conversation leaded by one of the 
teachers, informing them about the Potential Issues 
Database. After the conversation, students had one 
hour to fill a second template, reporting elements 
related to the consequences of the use of personal 
information that they identified as relevant for their 
concept. The second session of students performed 
the same activities, but at first, they fulfilled the second 
template without the support of the Potential Issues 
Database, and, later, they had the group conversation 
(Table 1). We compared the outputs (templates 
produced with and without the support of the 
knowledge contained in the Potential Issues Database) 


so to verify the usefulness of the conversation on issues. 


Step 4 - Selection of Issues and Discussion of 
Improvements. As a last step, we asked the students 
to select, for their own project, the issues they 
considered as most relevant and worth of discussion. 
We also asked them to list the improvements they 
would propose for their concept in a third template. At 
the end of the day, we collected student feedbacks in a 
questionnaire about the activities performed. According 
to the answers, the students found useful both the 


content and the discussion. 


4. Results of the Use of the Light Data 
Impact Tool 


The results of the workshop, together with some 
further experiments, confirmed the usefulness of the 
IAM to orienting designers toward improvements 

of their concepts, and to raise awareness about the 
consequences of their design choices. Furthermore, 
the workshop produced evidence of the effectiveness 
of the light tool, since: i) all the students stated that the 
lecture and the group discussion allowed them to find 
interesting correlations between the use of personal 


data and possible impacts; ii) the interactive exploration 
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of the Light Potential Issues Database [28], organizes 

in critical themes [20], reduces the complexity related 
to the amount of information allowing designers to 
nimbly access the knowledge saving time and effort; 

iii) the students were able to improve their projects 
considering aspects related to the privacy and rights 

of the user (pointing out the advantages of avoiding 
storage of unnecessary data, and the importance of 
giving users the control on data sharing to avoid the 
perception of being exposed — e.g. deciding to allow 
users to clog a camera while not used — Fig. 4), but 
also including aspects related to impacts that goes 
beyond privacy concerns such as avoiding overload of 
information through careful redesign feedbacks from 
the system (e.g. reducing the amount notifications, 
deleting data after 4 hours, and creating interaction 
summaries in the form of abstract artworks that 
aggregates data avoiding useless details and comparison 
between involved users — Fig. 5); iv) although many 

of the improvements focused on avoiding problems, 
others aimed at increasing features of the concept as 
result of discussing about opportunities given by the 
use of personal data. Working on eating experience for 
people with impaired taste [29], a team proposed that 
the service could consider user's food preferences to 
provide additional features. Another team, working on 
an automatic translator for slang and common language 
phrases [30], considered storing translated sentences to 


allow users to recall them when needed. 


5. Conclusion 


The validation activities and feedback from students 
provided useful insights about strengths and criticalities, 
supporting the Light Data Impact Tool and the IAM 
method approach producing evidences about two 
topics: First, the results of the identification of personal 
information involvement in the concepts, show that 
students struggled in understanding what personal data 
their concepts use, how the information flows work in 
their solution and what are the involved actors. They 
have little prior awareness about the role played by 
personal information in their solution, and about the 
criticalities and potentials. The introduction of a tool for 
representing data and their flows in the system, enables 
thinking and addresses the attention on this subject. 
Secondly, the students had difficulty on exploring the 


numerous issues and clusters of information contained 
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Fig. 4. (above) Tasteless’ [29] is a project of a neckless that imply 
image recognition for food experience. After the use of the tool, 
the students included the possibility to clog the camera while 
not used so to protect the user's privacy concept. 


Fig. 5. (below) ‘Cocoon’ [31] is a project of a bracelet aiming 


to improve physical communication between people in long 
distance relationships. The use of the tool led the students to 
carefully design notification mechanisms so to preserve meaning 


while avoiding information overload. 


in the Potential Issues Database; the knowledge 
generated by the research is huge and difficult to 

access. On one hand the observation and the students’ 
responses to a further online survey related to the 
activities and during informal interviews, point out the 
importance and usefulness of the development of the 
Light Potential Issues Database as a valid tool to support 
their awareness on both the use of personal information 
and its possible consequences. On the other hand, 

they demonstrate the need of agile ways to address the 
topic in the design process. Students faced their lack of 
knowledge about how and when personal information 

is implied in their designed solutions, and on potential 
impacts that the use of such information and the derived 
knowledge could have. This made clear that there is 

a need of critical thinking for the design of solutions 
implying the management of personal data. The design 


of tools that help designers in envisioning and foresee 


impacts demonstrates to be a valid contribution for the 
design culture to allow designers in creating responsible 
solutions that consider ethical and social issues in the 
design process and stimulate the conversation about 


responsibility in the design field. 
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Prosumeristic Publications: 


Abstract 


This article describes the design process followed in 
the making of a prosumerist publication. Alt*yd is a 
printed zine that goes beyond one-way communication 
with its readers. It takes into account the reactions 

of Instagrammers to its digital publication and using a 
program, incorporates it into its printed self. Inspired 
from the Open Source and Open Data culture, project 
alt*yd reflects upon the kind and medium of content 
most consumed today, and the relevance and possible 
adaptations of traditional media to keep in pace with 


emerging digital culture. 


Keywords 


Prosumerism, Internet Culture, Design Process, Print, 


Editorial Design 


1. Introduction 


In this article we present to you an experiment in 
editorial design, the output of which is a zine named 
alttyd. The experiment involved integrating social media 
as a programmed content source for a print publication. 
Simply put, we created a program that would convert 
Instagram profile data into printable publication spreads. 


We believe writing about this editorial design process 
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alt--yd 


is of importance as our experiment is potentially one 
of several precursory examples exploring the future of 


print publication. 


2. Background 


The National Institute of Design is India's premier 
design education, service, and research institution. 
Every year at the school, hundreds of students sit for 
the annual campus recruitment event, colloquially called 
the ‘placements’. There are two kinds of opportunities 
at the event: for students seeking graduation thesis 
internships and for graduates seeking full-time jobs. 
Those who fit the cookie-cutter requirements of the 
industry are selected and often offered large starting 


packages. 


The 'placements' is a good platform to meet potential 
employers if what the student seeks is to work in 
these said companies. However, most students are in 
the process of figuring out what they want to do and 
find themselves to not fit the demands of the campus 
recruitment companies. Students whose portfolios 
do not adhere to the traditional interpretations of 
their disciplines find it difficult to market their design 
approach and practice. Furthermore, younger design 
students inadvertently begin looking up to the industry 
norms thereby preventing themselves from exploring 


the field independently. 


Fig. 1. (left to right) 
The Instagram post, 
the program, the 
generated spread, the 


printed book 
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In light of these micro-conflicts, discussions arose and 

a shared desire was expressed by the authors to create 
a platform that showcases alternative projects and 
celebrates experimentation. The aim was to highlight 
projects that touch upon non-mainstream subjects and 
methodologies but often go unnoticed. Therein the idea 
of a publication documenting such projects emerged. 
The name alttyd is a take on the Young Designer (YD) 
book, a compilation of all the graduation projects, 
published every year by the National Institute of Design 
(NID). 


3. Intent of the Zine 


Alttyd is a curation of graduation thesis projects 

done at NID that explore emerging design practices 
and futures of the country. By showcasing alternative 
possibilities of graduation projects, alttyd aims to be a 
source of inspiration for budding design students and 
is intended to be a statement to the design industry on 


the fresh possibilities that the field provides. 


4. An Experiment in Editorial Design 


Publication has traditionally been a one-way 
communication platform where the printed material 
communicates to the reader. The authors decided to 
ask- what might a book look like if every reader, in some 
way, could contribute to the published content? In the 
ever-connected world of the internet, how might one 


achieve that? 


Hopping from the concept of shared Google docs to 
crowdsourced[1] content, the authors landed on the 


concept of Prosumerism[2]. 


5. Prosumerism 


A Prosumer([2] is a person who produces as well as 
consumes a product. In the context of the early 21st 
century, the term 'Prosumerism' blurs the meaning of 
the term ‘product’ itself. On almost all digital platforms 
today, the consumer of the digital content is often 

also the producer of the said content. A give and take 
of data to and from the people runs most systems 
today. Interesting results can be seen if the concept of 
prosumerism is applied to other, traditionally one-way 


media. 


6. Prosumeristic Publication: A Note on 
Production Methodology 


What might we get if we ‘Ctrl+P’ an Instagram profile? 
Given its wide reach and ease of use, Instagram was 
chosen as the social media platform to display alttyd. 
An Instagram page (@altplusyd) was created for the 
project. The primary content of the zine- a curated 
collection of provocative projects was regularly updated 
on the page. The aim of the page was to elicit dialogue 


and reactions to the digitally published projects. 


An application was custom developed in the Processing 
Development Environment (PDE)[3] to extract all the 
content of the Instagram page and convert it into print 
ready spreads. This application exploited Instagram data 
available to its users in the form of JSON[4] files. These 
files helped with the extraction of two kinds of data- 1) 
The metadata: the project title and the department was 
extracted from the hashtags, 2) Project information: the 
project description, comments and likes were pulled 
from the post. The consequent spreads therefore 
contained not only the projects but also the reactions 


and comments they had garnered on Instagram, 
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ensuring that the same audience who would read the 
book in the future could contribute to its contents in 


the present. 


The additional standard pages of a publication such as 
the title, conclusion and print notes were created in 
InDesign and added to the generated PDF of the main 
text block. The technique of foiling was used for the 
cover page, each cover was foiled by hand, giving each 
copy a unique appearance. The cover and the text block 


were then center-stapled and cut. 


The first edition of alt*yd was available at the 39th 
Convocation Ceremony of the National Institute of 
Design, 2019. Every printed edition thereafter will be 
different from its predecessor due to its ever evolving 


content. 


7. Freedoms and Constraints of the Project 


The authors had total freedom on the design and 
production of the project. Constraints were few but 
hefty. The project was conceived on December 1, 
2018 and had to be delivered by January 10, 2019. The 
entire timeline for conceptualization, prototyping, and 
production was all within a month. The budget of the 
project was restricted to 5000 INR, i.e. around 70.44 
USD. 


8. Reflections 


The eye is well acquainted to how a book reads. So 
much so that the mind forgets that the book is the way 
it is, owing to the freedoms and constraints of design 
and production dating back to the era of the printing 


press. 


A prosumeristic publication comes with its own 
quirks on how it reads. These quirky elements are not 


entirely new. It was observed that they merely seemed 


misplaced to the eye in the context of a printed booklet. 


The following are the observations and feedback of the 


readers of alttyd as well as the authors: 


1. The emoji “W45’ (heart icon + a number) is a familiar 
symbol. Most are aware of what it signifies in the 
digital world- that 45 people have ‘liked’ the post. Yet, 


finding it on printed paper was a new experience. 
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Fig. 2. Production 
cycle of alttyd 
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2. Proper nouns were no longer just words. Any name 
which had an Instagram profile to it was tagged with an 
‘@’. This is a common practice in the digital world, yet, 
seemed uncommon when every name was followed by 


the Instagram counterpart of it. 


3. Comments on the digital platform are another 
regularly occurring phenomenon. Yet, printed 
comments seem unusual, even more so when on reading 


them one realizes that they could be written by bots! 


4. The usage of ‘#hashtags’ to control the content flow 
from the Instagram posts to the publication- Since 

not all posts could be a part of the publication due to 
certain editorial decisions and constraints, a new system 
was devised to control this content flow using hashtags. 


These hashtags helped the custom-developed program 


Fig. 3. Printed copy 


of alt--yd, first edition 
(2019) 


Fig. 4. Example of 
a printed comment, 
potentially from a 
bot account, alt+yd 
(January, 2019) 


to identify the posts that were to be fully incorporated Citations 


in the publication, the posts to be included partially 
as well as those that were supposed to be excluded 


completely. 


1. Howe, Jeff (June 2, 2006). "Crowdsourcing: A Definition". 


Crowdsourcing Blog. Retrieved January 2, 2013. 


2. Toffler, Alvin. The Third Wave. Bantam Books, 1990. 


5. Comments that tagged other people, thanked 
someone or were emotionally charged, suddenly 
seemed to have relatively more gravitas in a printed 
publication than online. An indication of how we still 


take printed content more seriously than digital? 


9. Conclusion 


This zine explores a future of print publication where 
the boundary between the digital and the physical 
medium is blurred. The physical is not merely a tangible 
copy of the digital. Rather, the physical could borrow 
from the evolving reactions to the digital, making 
'reactions' another form of content. With a future- 
oriented outlook, the approach with which alttyd was 
made asks the following questions: 

‘In what ways might the boundary between the digital 
and the printed be blurred?’ 

‘How might ‘open source’ be interpreted in the world of 
print publication?’ 

‘Can a reader contribute to a book even after it has 


been printed?' 
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of Things 
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In an age where computers are shrinking in size, 
sensing is cheap and data abundant, the author 
investigates the concept commonly known as the 
Internet of Things. The author dissects the 
Internet of Things from social, cultural, economic, 
technological, political and ethical standpoints. 
Through prototypes, the author attempts to gain 
insight into how the Internet of Things technology 
is intertwined with social, cultural, ethical and 
political aspects of human life and how the 
technology gains relevance when viewed from 
these perspectives. 
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Swimming Coach: An Immersive Swimming 


Learning System 


Abstract 


Swimming is a common activity to keep fit. To assist 
hydrophobic users to overcome the fear of learning 
swimming, and to give full play to the advantages of 
land swimming practice, we propose a swim learning 
system which includes VR system, wearable devices and 
frame structure to provide users with an immersive and 
interactive learning experience of swimming without 
water. Users’ real-time postures can be detected 

by using the body gesture recognition system, while 
visual and haptic feedback are also available to users. 
Preliminary investigation shows that the proposed 
system has demonstrated potential in the field of 


swimming learning. 


Keywords 


Swimming Learning, Virtual Reality, Immersion, Head 


Mounted Display, Gesture Recognition 


1. Introduction 


Swimming is a popular activity that offers physical and 
mental benefits. It ranks as one of the sports activities 
that people like most [1, 2]. People learn to swim not 
only for health and recreation [3] but also for self- 
rescue [4]. However, swimming is a complex sport 


that requires a flow of quick, smooth and accurate 
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movements, so swimmers need external guidances and 
constant practices to build muscle memory [5] by land 
swimming practice. Due to the lack of related scientific 
learning system, beginners are hard to overcome the 
fear of water. Besides, they do not receive immersive 
swimming feedback, which do not suffice for fast 


improvements. 


Thus, we propose "Swimming Coach," an immersive 
swimming learning system. The system can help 
beginners to learn to swim with an immersive and 


interactive swimming learning experience. 


2. Related Work 


Some researchers created virtual immersive 
underwater environments to mimic the real visual 
feeling. For instance, Shogo Yamashita et al. designed 

a cave system with multiple stereoscopic projections 

on surrounding acrylic walls, providing swimming 
environment with swappable scenes, such as coral reefs 
and shark cage diving [6]. Similarly, Lisa Blum et al. 
presented a system using augmented reality techniques 
to simulate a regular swimming pool with virtual objects 
[7]. The above works made users feel that they were in 
water environments visually, though researchers did not 


provide other sensory experiences. 


Some works used certain physical structures to simulate 


Fig. 1. A user is wearing 
HMD and experiencing 
Multi-scene swimming 


learning. 


Fig. 2. The inflatable air 


cushion. 


swimming in water under the condition of waterless. 

In the system of Tzu-Pei Grace Chen et al., the user 
was suspended in a real swimming apparatus which 
simulated the swimming condition in a virtual Pacific 
Ocean environment [8]. Dhruv Jain et al. also used the 
suspended system, but they placed an inflatable airbag 
under the user's belly, which allowed the user's body to 
rise and fall in sync with breathing [9]. These two works 
allow users to move limbs, but they do not include 


swimming poses in different scenes. 


To create a real experience for users, and to let users 
realize the transition from not being able to swim to 
swimming, the feeling of immersion and presence of 
users should be taken into account when designing the 


swimming learning system [10]. 


3. System Design 


Based on previous works, we designed "Swimming 
Coach", which includes four components: body 

support prototype, Head Mounted Display, multi-scene 
swim learning application, as well as human gesture 
recognition and matching module. Our system can help 
swimmers to learn swim in various swimming scenarios, 
and learn multiple postures (competitive swimming and 


practical swimming). 


— — € 
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3.4 Swimming Support Prototype 


We designed and made a body support prototype. In 
this support prototype, a support frame is attached to 
the user by a harness which allows the user's hands and 
feet to move freely. Notably the support frame can be 
rotated so that the user can learn how to swim in any 


vertical angles. 


To simulate water resistance, we designed a wristband 
(Fig. 1), which is connected to the safety belt through 
an elastic band. What's more, to simulate the feeling of 
floating in the water, we added an inflatable air cushion 
on the support prototype (Fig. 2). The inflatable air 
cushion is connected to an inlet air pump and an outlet 
vacuum pump through the air pipe. When users are 
learning, they can experience the feeling of flotation 
through the expansion and contraction of the air 


cushion. 


3.2 (Head Mounted Display) HMD 


To enhance the feeling of immersion and increase the 
interaction of "Swimming Coach" provided by tactual, 
visual and auditory feedback, we develop the HMD 
(Fig.1). It includes VR glasses and a breathing mask with 
an airbag. 


We set the completion time for each set of actions, 


TN @® Inflatable Air Cushion 


© Vacuum Pump 


Design and Semantics of Form and Movement 225 


during which the user needs to complete the 


corresponding action. 


In the VR glasses, the user's main field of view is his or 
her first-person view (Fig.3). The upper left corner of 
the interface is the matching between the image from 
the third-party perspective and the correct posture. 
If the user's action is inconsistent with the standard 


action, the interface will present voice prompt. 


The airbag of the breathing mask (Fig.4) is prompting the 
user to inhale and exhale correctly while swimming. The 
time interval between airbag expansion and contraction 
is consistent with the standard swimming breathing rate. 
We use this device to prompt the user to inhale and 


exhale correctly while swimming. 


3.3 Human Gesture Recognition and Matching 
Module 


Based on Kinect and Unity, we designed a set of human 
gesture recognition and matching module, which can 
identify the user's bone nodes in real time, and then 
match the recognition results with the bone nodes 

of standard actions to determine whether the user's 
swimming action is correct or not. VR glasses will also 


give users corresponding hints. 


3.4 Multi-scene Swimming Learning Application 


According to previous research, locations of drowning 
by different age groups were categorized into the 
swimming pool and open water (such as a lake, river, 

or pond) [11]. In our system, we use multiple scenes in 
VR vision to help users practice swimming in different 
kinds of water. We use Unity to create our scenarios, 
such as the swimming pool and ocean (Fig. 5). Besides, 
we design an oxygen value (green bar) and a health value 
(blue bar). This setting can simulate the real status of 


swimmers in the water and provide data for the HMD. 


4. Evaluation 


20 volunteers (female 7, male 13) with an average 
height of 173 cm (SD=9.2) and an average weight 

of 58 kg (SD=11.7) were recruited from Zhejiang 
University, China. In order to test the effectiveness 
of the system. (e.g. In the experiment, the volunteers 


used the Swimming Coach one by one for 30 minutes 
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Fig. 3. A scene from the 
HMD when users learn 


freestyle swimming. 


under proper instructions from the researchers. The 


volunteers tested two different swimming scenarios). 
After experiencing this system, the volunteers were 
asked to complete a questionnaire [12] and then 
interviewed concerning their feelings and feedbacks 
about the swim learning system. The data were later 
analyzed using a chi-square (x2) test [13]. The reported 
overall rating of presence across all participants was 
4.79/7 (SD=1.59). It indicated that they were positive 
about the system and felt immersed. 

Meanwhile, according to the interview, 17 of 20 
volunteers can learn the basic swimming posture, 3 
volunteers said the discomfort caused by the safety belt 


could affect learning to swim. 


5. Conclusion and Future Work 


In this paper, we present an immersive and interactive 
VR swim learning system, a combination of HMD, 
human gesture recognition system and multi- 

scene swimming learning application. Through the 
experiment of 20 participants, the system proves to 
be effective to swimming learners. However, we are 


aware that certain problems do exist. For example, 
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Fig. 4. The breathing mask. 


Fig. 5. Multi-scene: 


swimming pool and ocean 


the swimming movements of participants often made 


wrong movements, leading to the massive warning 
voice prompt from the system, which might impair the 


immersion experience of the users. 


In future work, we will improve the body support 
prototype, and change the form of error reporting 
from voice prompts to color changes of characters in 
VR scenario, and conduct more tests to perfect our 
"Swimming Coach". Besides, we plan to upgrade this 
equipment into a more general device, which can be 


used for parachuting, gliding and other sports. 
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Designing Transparent Collaborations — 


Weave 


Abstract 


We are eager to converse about the approaches we 
may need to take when designing for complex intelligent 
ecosystems, their meaning, experience, and ethics, 

but | would like to invite you to a conversation on 
rethinking and redefining how we and our tools in our 
collaborative processes could better handle complexity, 
ethics, and transparency among each other before 
intelligent ecosystems are brought into the picture. 
After all, the design solutions we propose are really 

a collection of our lenses, our framing capacities, and 
the quality and ethics of our decisions, all derived 
significantly from our collaborative practices and the 
tools we use — and the collection of decisions from 
which they are made. 

This document contains detailed information about 

the creation of Weave. Weave brings designers and 
developers together at the beginning of product 
development, enabling them to co-create the system 
architecture, thereby adding clarity and transparency 
throughout the collaboration. The goal is more 
effective, more efficient processes, and a deeper, more 
trusting relationship among collaborators. 

Weave was created through a detailed study of the 
current design and development tools, primary research 
with Designers, Developers, and Project Managers, and 
four rounds of testing with 39 participants who follow 


agile processes. 
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1. Overview 


1.1 Current Practices 


Currently, the majority of the collaboration that takes 
place between designers and developers begins after 
most of the foundational design and implementation 
decisions are made. Currently, collaboration begins 
once a designer hands over a final interface prototype 
to the development team for implementation. This 
pattern creates a workflow that is full of uncertainty, 
paving the way for misunderstandings and creation of 
assumptions to understand the decisions made by both 
parties. This is where both parties often end up with 


outcomes they did not intend. 


Organizations encourage designers and developers 

to work together in the same place, at the same time 

to help develop a shared understanding in designer- 
developer collaborations. However, this is not a viable 
option when the two work remotely. Some companies 
spend resources, time, and money to fly a designer 
from one office to another simply to converse in person 


with the development team about UI/UX specifications, 


and decisions. However, this approach is, once again, 
not always viable, and it typically happens once, when 
a deliverable deadline is approaching. Weave proposes 
a collaboration process that is not only consistent and 
continuous but also meaningful and comprehensive, 
where the focus of the collaboration is on the 
foundation of decisions rather than the surface-level 


actions that lead to the decisions. 


In the design process the act of prototyping has 
multiple purposes. Purposes of prototyping consist 

of, evaluating and testing the design (both within 

the team/internal and with the users, aka, usability 
testing), communicating the idea/design with others 
(includes selling the idea to the business), setting 
design priorities, and clarifying production costs and 
issues. Even though the Design industry, specifically 
firms that build interface prototyping tools, have been 
showing that they are thinking about collaboration and 
workflow, they, currently, do not encourage or help 
designers and developers to understand each others' 
workflows and priorities and tend to find very surface- 
level solutions that do not consist of any variety or at 
least the appropriate variety. When prototyping, the 
highest level goal of the designer and the developer is 
reaching the production stage. Also, current interface 
prototyping tools, lack in consistency when it comes 
to features offered, and purposes for use. For instance 
upon selecting a prototyping tool, a designer might 
have to predict any potential design changes that may 
come up in the process in the future, such as importing 
an interactive graph, because only a few tools may be 


capable of doing this. 


1.2 Project Goals 


The goal of this project is to bridge the gap that 
currently exists in Designer-Developer collaborations 
in order to improve their relationships. Improving 
Designer-Developer relationships is made possible by 
increasing their understanding of each others' approach, 
which can, in the long-term, increase the trust that 
exists between them. Increasing trust in Designer- 
Developer relationships is made possible by ensuring 
that they can truly collaborate in a workflow which can 
reveal where they are aligned and where they are not 
aligned with each other early on in projects by working 


together to make determining decisions, before either 


one has individually made determining design and 


implementation decisions. 


2. Project Framing and Convergence 


2.1. Initial Hypothesis 


The following formed the initial hypothesis in this 


project: 


1. Designers and Developers do not have a deep 
understanding of the logic that exist behind each others' 
decisions. 

2. Designers and Developers are not able to actively 
propose changes to each other's' work. 

3. The meaning of collaboration has changed today. In 
the Designer-Developer collaborations there seem to 
be more attention to details rather than conveying the 
framework of what their decisions, such as the logic of 


their constraints that will shape their goals. 


Initial Hypothesis: Therefore, if there is a common 
ground on which Designers and Developers can 
collaborate, where both know how to propose changes 
to, then a they can begin to build a foundation for their 


collaborations throughout the course of projects. 


3. Problem Definition 


3.1 Problem 


The major problem with the current workflow of 
Designers and Developers, mentioned above, is that the 
initial collaborations tend to stay on the surface level 
and do not go deep. Designer-Developer collaborations, 
currently, become more detailed and more frequent 
once the Developer receives clickable prototypes from 
the Designer and needs to understand and make sense 
of them in order to implement them. The point at 
which Designer-Developer collaborations become more 
detailed and frequent is a problem because this is when 
most of the design decisions and perspectives have been 
made and formed. When a Designer communicates her 
design decisions with a Developer, she is communicating 
three layers of information all at the same time. The 
layers being communicated are, 

1. The architecture of the system 

2. The user interface specifications, such as colors, 


fonts, and dimensions 
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3. The specific user interactions 

Communicating three layers of information all at the 
same time with Developers is a major problem, because 
it divides their attention, deferring the recognition of 
inconsistencies that exist in any of the layers mentioned 
above. Currently, Developers face a great deal of 
difficulty trying to understand the three layers of 
information that are communicated to them. Therefore, 
a lot of back and forth occurs between design and 
development, where many parts of the design decisions 


in each layer are changed as the result. 


4. Value Proposition 


4.1 Benefits of Weave 


The most important goal of this product is to help 
change some of the current ways of thinking with 
regards to prototyping tools and the process of 
collaboration from design to implementation as a 
whole. It is important to note that, today, there are no 
products in the market that are looking at this issue 


with the level of depth Weave is addressing it. Weave: 


1. Views collaboration and conversation as a two- 
sided process, where not only developers have the 
opportunity to learn about designers' logic and 
approach, but also designers have the opportunity to 
learn about developers' logic and approach. 

2. Allows both designers and developers to work 
together to propose changes to the system, and not 
only collaborate by giving each other written feedback 
or feedback that focuses only on constraints. 

3. Provides an opportunity for both designers and 
developers to collaborate in their workflows as part 
of their natural process, because the foundation of this 
collaboration is rooted in the co-creation of system 
architecture. 

4. Provides an opportunity for both designers and 
developers to document their entire process on 
contextual levels so that they can have access to 
everything in the future when needed. 

5. Uses only one of the information layers to begin the 
designer-developer collaborations, therefore it is able 
to focus on a less complicated collaboration process, 
where the development of trust can be highlighted. 
[1-13] 
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5. Primary Research 


5.1 Details and Findings 


Research was conducted with a total of 15 Designers, 
Developers, and Product Managers who work at 
technology-based, agile companies in order to learn 
about the current problems that exist in the workflow 
and collaborations, as well as to test the hypothesis 


mentioned earlier. 


Some of the most important insights gained from the 
research conducted are: 

1. Designers and developers both expect each other to 
add knowledge about their domain. 

2. They express expectations after something has 

gone wrong, exhibit lack of ability to avoid deferred 
recognition of misunderstandings and disagreements, 

a lack of motivation to convey key information, and 
documentation. 

3. In-person communication is their most preferred 
type of communication. 

4. Design sprints are often expanded causing 
development sprints to compress, where developers 
experience a great deal of stress. 

5. Designers experience a lot of frustration when 
needing to revise design decisions a few times after they 
have already finalized them. 

6. Developers often end up making most of structural 
and architectural decisions that UX designers need to 


make. 


6. Usability Testing 


6.1 Details and Findings 


There are four phases of usability testing, all of which 
were conducted with a total of 39 designers and 
developers. The four phases of usability testing were 
conducted in order to 1. test the first design concept, 2. 
understand users' mapping logic and behaviors, 3-4. and 
to test the interface flow and its learnability. Prior to 
testing phase #2 the Unified Modeling Language (UML) 
were studied in order to understand how current 
mapping systems work, and what their logics are. UML 
is a standardized modeling language enabling developers 
to specify, visualize, construct and document artifacts of 


a software system. [14] 
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The overall feedback are as follow: 


1. The mapping of systems is necessary and where most 
of the problem lies, because no mutual understanding 
and agreement between the designer and developer is 
formed at this step. 

2. Limiting the designer’s choices may be possible for 
more explicit development frameworks, mainly with 
interface frameworks such as Material Design or iOS, 
however this is not technically possible with interaction 
frameworks because they tend to be loose, and 
therefore hard to be used for drawing any comparisons 
with visual designs of a designer in order to detect any 
“design errors”. Limiting a designer’s choices may be 
efficient, but it will not encourage any collaborations 
and is not ethical. 

3. It is crucial to communicate the activities that take 
place in a system as well as the states that it is in 
throughout its design process. 

4. Add a motivation factor for designers and developers 
to continue to collaborate. 

5. Ensure smoother user adaptability by adding multiple 
initiation points to the system, i.e. allowing users to 
enter the system from anywhere they are in their 
workflow. 

6. Focus more deeply on “agreement” and 
“disagreement” functions. 

7. Add more user control by providing brief note-writing 


for conveyance of messages. 


7. Design Solution 


7.1 Details and Flow 


Weave is a platform that enables Designers and 
Developers to begin their collaboration at the beginning 
of projects, before any design decisions have been made, 
by co-creating the system architecture. Designers can 
use Weave as a plugin within interface prototyping 


tools, and Developers can use Weave through its web 


app. 


There are five major phases that the user of Weave 
goes through, they are: 

1. Weave Activation: This is point at which, the designer 
activates Weave as a plugin in the interface prototyping 
tool that she/he is using. 

2. Weave Initiation: At this point the designer begins 

to create a user scenario, which Weave, using natural 
language processing, will then turn into a map that 
represents the initial system architecture. The designer 
inputs a user scenario by inputting individual user 
actions, broken down to verbs, system parts, and 
functions. For instance, “signs in to the app”. Signs 

in: verb; app: system part. Or for instance, “presses 
Reserve’. Presses: verb; reserve: function. The designer 
can further edit the map that has been created by 
adding more actions to it, replacing or deleting actions 
from it. 

3. Collaboration Initiation: At this point the designer 
will publish what she has created, allowing her to 

share it with her collaborators. Once shared with 
other collaborators, they, specifically the developer 

can propose changes to the architecture that has been 
created, either by adding more actions under each map, 
or by adding variations to each map beyond the user 
scenario that has been created, in order to encourage 
progress to be made in the creation of the entire 
architecture so that the developer can begin to become 
more certain about her next steps and to begin to 
frame her code. 

4. Collaboration Progression: At this point both the 
designer and developer collaborate continually by 
responding to each others’ proposed changes. 

5. Documentation: At this point designer’s responses 

in the user scenario section, publishing section, as 

well as the progress made on collaborations through 


adding attachments, notes, expressing agreements/ 
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disagreements, and creating integrations with 


other project management tools, all result in the 


documentation of the process of a project and 


collaboration. This is shown in Figure 1. 
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Huxley: Intelligent Book as Essentialist 


Artefact 


Abstract 


Huxley is a system that uses measurement of a user's 
focus and interest throughout the day in order to 
alter a ‘smart-book’ in their home space to make it 
more engaging. By tracking user interaction with the 
book over time, the home environment also learns to 
alter peripheral cues in imagery, lighting, and sound— 
controlling the character of the space and the visual 
salience of the book— in order to encourage its use. 
Huxley is an environment that is informed about user 
goals and works to encourage better decisions and 
deeper focus in line with those intentions. It is an 
artefact of an essentialist, attention-centered design 


philosophy. 


Keywords 


Attention, Salience, Smart Home, Persuasion, Cognitive 
Affordance 


1. Introduction 


A decade ago Nicholas Carr first raised the alarm 
about negative impacts of the information age on 
human cognition and phenomenology. [1] It is now a 
chorus in the popular tech literature—our focus and 
concentration has been hijacked by our information and 


communication technology (ICT). Carr argues that the 


rapid, non-linear navigation of a hyperlinked internet 
has already predisposed us away from deep, focused 
reflection on singular topics. Ad-driven business 
models—which necessitate addictive design patterns 
and a diversion of users away from their intent— 
intensifies a trend towards fractured attention that is 


native to the medium itself. 


Unfortunately, the research bears this out. Most 
websites are visited for less than 15 seconds; all 
websites average less than a minute. [2] Moreover, 
hyperlinked content detracts from learning outcomes 
while increasing interaction time. [3] Smartphone 
research is more directly disheartening— Ward et. 
al.'s 2017 work [4] showed that having one’s phone in 
the visual periphery significantly reduces one’s available 


cognitive capacity for other tasks. 


Against the backdrop of behavioural economics, we can 
frame the internet and the smartphone another way— 
their structures impose a state of constant decision- 
making on the user. These designs force a regular re- 
evaluation of the task at hand; with instant and limitless 
access, they also prevent us from full commitment to 
any one task. We know from the work of psychologist 
Barry Schwartz that too many choices lead to 
paradoxical dissatisfaction. [5] The ability to renege on 
our selections also leads to greater unhappiness, despite 


our tendency to keep options available. [6] 
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How should this knowledge impact future design? 
While there are many open questions about the nature 
of the relationships between an artefact's symbolic 
information, conceptual model, [7] and its cognitive 
effects, it is clear that we certainly should avoid 
conceptual models and addictive design practices that 
are linked to cognitive overhead, and likely should avoid 
the corresponding symbolic information that has been 
rigorously paired with fractured attention over the last 
decade. More research needs to be done to answer 


many important questions: 


* How tightly coupled are conceptual models 
and cognitive load? Is the relationship binary or 
continuous? (If your phone decreases your focus, will a 
similar phone? Any phone?) 

* Have we learned a direct association between 
symbolic information and cognitive state? Do new 
conceptual models alter it? (Does a ‘phone form’ induce 
high cognitive load, even with no phone functionality? Is 
this reversible?) 

* What aspects of these devices are most damaging to 
focus? (How much does the type and nature of a device's 
functionality versus addictive design choices like variable 
reward structures, badges, and infinite scroll contribute to 
cognitive load?) 


In the spirit of ‘Calm Technology’, [8] Huxley is designed 
against the backdrop of these provocations, both as 

a tool for inquiry and as an example of attention- 
aware design. Other theories of design offer notions 
of psychological pairing between an object's sensory 
features and a user's emotional response to it; [9] 

we submit this idea logically extends to the user's 
resting attentional demand when using the object as 
well. We believe the psychology supports a move 
toward essentialism [10] in design—environments and 
artefacts whose choice architecture promotes fewer, 
longer, deeply engaged experiences in line with a user's 


priorities for themselves. 


There is a rich confluence of cause-and-effect to 


language based on behavioural economics and cognitive 


models of deep engagement. 


2. Huxley 


The Huxley system is composed of three main parts: 

a chrome extension to track focus and interest during 
the day, the smart book artefact itself, and a peripheral 
lighting and video system to alter the affective quality of 


the reading environment and book salience. 


2.1 Interest Tracking 


The first part of Huxley is a chrome extension that runs 
in the browser on the user's primary work computer, 
collecting and storing data including the number of tabs 
and windows open, the websites visited, the number 

of task switches, and overall active and idle behaviour. 
This information is processed to give a measure of focus, 
the duration worked, and the topics the user is engaging 
with based on Google queries, time on 'timewasting' 

or ‘productive’ domains, task switching, and other 


browsing behaviour. 


2.2 Smart Book 


Huxley (Fig. 1) is a book with three e-ink displays, a 
single button, and a charging USB port. It behaves like 
a book—its spine and cover display the title and cover 


of the book contained within, and once opened the 


disentangle given the current state of screen-based, 


Fig. 1. The Huxley Smart Book, showing updating e-ink displays on the cover, spine, and 


multifunction artefacts. We hope Huxley will spark inside. A USB port on top is available for charging, one button is available for turning pages. 


fruitful conversation about the cultural and aesthetic Huxley only updates when the user is away and the current book has been completed or 
implications of the Attention Economy’s addictive design ignored. Its selection is based on the measured interests and affective state of the user at 


tradition and move us towards an empirical design work. 
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internal e-ink page can be turned with a button press. 
There are no indicators of battery state or wireless 
connectivity, no menus, and no way to change the book 


contained within. 


Instead, Huxley broadcasts an API over the local 
network—once loaded with PDF/EPUB files, it can be 
directed to become a random selection (favoring new 
books) or become a relevant book based on provided 
queries (using Doc2Vec embeddings as a similarity 


metric). 


Huxley was created with a few motivating principles in 


mind: 


Minimized Choice. Good technology helps curate 
our world and eliminate the ill effects of the paradox of 
choice when we're trying to focus. There is no ability 
with Huxley to summon any book in the universe, and 
no leaving the option in front of you behind. 
Permanence. Huxley only switches when the user is 
away. The interaction is designed to make it feel like a 
single, permanent object that doesn't have the ability to 
access any external information. 

Simplicity. The conceptual model is very simple-- it 
works like a book. It doesn't have WIFI or battery 
indicators. The affordances are equally simple—one 
button, one possible interaction. 

Evocative Aesthetics. Huxley evokes the conceptual 
model of a book without false affordances (it's clear 
there are no pages to turn). This helps frame the 
interaction as ‘book-like’ instead of ‘screen-like’. 
Increased Cost. Many of our services allow us 
ephemeral access to unlimited media. The ‘zero-cost’ 
model (time, space, money, and effort) disincentivized 
attribution of value to any included content. Huxley 
takes up space in the home, a physical cost that 
ascribes worth to its content and incentivizes focused 


engagement. 


2.3 Peripheral Guiding and Intelligence 


Users prefer different lighting for different tasks; [11] 
we hope that priming the mood also encourages the 
associated activity. Increasing a behaviour like reading 
usually involves more frequent 'triggers' to perform the 
action. [12] By manipulating book salience, we aim to 


draw the user's eye to the book more frequently— the 


more they notice and consider it, the more likely they 


are to engage with it. 


Lighting, visual imagery, and book spotlighting are 
controlled using a custom video looping/DMX lighting 
control server running on a Raspberry Pi. Though 
the system is capable of a full range of expression, for 
simplicity we've devised two basic conditions (Fig. 2). 
The system uses a running average to bucket focus 

and time worked from the interest tracker into four 
categories (i.e. high focus, long day). Each of these 
categories is treated separately with a Bayesian Bandit 
optimization over lighting choice and book relatedness to 
work queries. We seed the system with common sense 
priors— for example, we expect long focused days to 
benefit from a calming off-topic book, short focused 
days to benefit from a high-energy related book, etc. 
By monitoring page turns, we update our model each 


day to choose the best lighting and book pairings to 


encourage use fora given user state. 


Fig. 2. Left, a high-energy scene of a city, with blue lights and a 
white spotlight on the book. Right, a warm scene with a fireplace 
and without book spotlighting. Mood and book salience are 


selected based on the Focus Tracking extension. 


3. Discussion and Next Steps 


Huxley is an adaptive system that nudges you towards a 
deeply engaged reading experience through peripheral 
cues, content selection, and artefact design. Its design 
principles are derived from essentialism in reaction to 
the current paradigm of attention hacking. We plan 

to study this interaction and quantify the its effect on 
the number and depth of book interactions compared 


with books, e-readers, and iPads in situ. We hope to 
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disentangle the influence of conceptual models and 
symbolic information on deep engagement—mostly 
learned, modern associations with ramifications for 


future design. 


We also plan to introduce eye tracking to explicitly 
quantify saliency's effect on behaviour and attempt 

to quantify the benefits of peripheral cues, alongside 
richer measures and models of user state. This will pave 
the way for more accurate and sophisticated analysis 

of dynamic causal intervention, and lead to a more 


effective overall system. 


4. Conclusion 


How do we design systems that increase our likelihood 
of doing the things we aspire to more frequently and 
with deeper focus? Can we marry the best of iPads to 
books, Spotify to records, or email to letters? 

Huxley was created with an understanding of the 
inherited problems of attention economy design, with 
a goal of promoting deep engagement. |t attempts 

to preserves the value of connected information 
technology, and behavioral nudging without the 
negative cognitive impacts. It is an artefact of a design 


philosophy built for focus. 


Huxley strives to feel like a close friend selected a single 
book for you every night after work; not an open portal 
to limitless options. We hope more technology moves 
towards essentialism and away from the fractured 
attention characteristic of the modern technology 


landscape. 


References 


1. Carr, N. (2011). The shallows: What the Internet is doing to 
our brains. WW Norton & Company. 

2. Liu, C., White, R. W., & Dumais, S. (2010, July). 
Understanding web browsing behaviors through Weibull 
analysis of dwell time. In Proceedings of the 33rd 
international ACM SIGIR conference on Research and 
development in information retrieval (pp. 379-386). ACM. 

3. Niederhauser, D. S., Reynolds, R. E., Salmen, D. J., & 
Skolmoski, P. (2000). The influence of cognitive load on 
learning from hypertext. Journal of educational computing 


research, 23(3), 237-255. 


236 Design and Semantics of Form and Movement 


Ward, A. F., Duke, K., Gneezy, A., & Bos, M. W. (2017). Brain 
drain: the mere presence of one's own smartphone reduces 
available cognitive capacity. Journal of the Association for 
Consumer Research, 2(2), 140-154. 

Schwartz, B. (2004). The paradox of choice: Why more is 
less (Vol. 6). New York: HarperCollins. 

Gilbert, D. (2009). Stumbling on happiness. Vintage Canada. 
Norman, D. A. (1999). Affordance, conventions, and design. 
interactions, 6(3), 38-43. 

Weiser, M., & Brown, J. S. (1996). Designing calm technology. 
PowerGrid Journal, 1(1), 75-85. 

Colombo, S. (2016). Dynamic Products: Shaping Information 
to Engage and Persuade. Springer. 


. McKeown, G. (2014). Essentialism: The disciplined pursuit of 


less. Currency. 


. Zhao, N., Azaria, A., & Paradiso, J. A. (2017). Mediated 


Atmospheres: A Multimodal Mediated Work Environment. 
Proceedings of the ACM on Interactive, Mobile, Wearable 


and Ubiquitous Technologies, 1(2), 31. 


. Fogg, B. J. (2009, April). A behavior model for persuasive 


design. In Proceedings of the 4th international Conference 


on Persuasive Technology (p. 40). ACM. 


Yuyu Lin!, Kai Zheng!, Lijuan Liu!, Yang Chen’, Jiahao Guo!, Shuo Li', Cheng Yao!, Fangtian Ying? 


"Zhejiang University, Hangzhou, China 
?Hubei University of Technology, Wuhan, China 


(linyuyu, hey. yahei, liulijuan, sonnechen, jiahaoguo}@zju.edu.cn, 


leesoo_shuo@outlook.com, Yaoch@zju.edu.cn, yingft@gmail.com 


OlfacEnhancer: A Vision-Based Scented 
Necklace for Cross-Modal Perception and 
Olfaction Augmentation 


Abstract 


Smell disorders are common, which can impair the 
quality of life. In this paper, we propose OlfacEnhancer, 
a pendant for hyposmia patients, which augments 
real-world olfactory perception synchronizing visual 
information. We present the compact hardware 

design that can release seven different scents and the 
implementation of a machine-vision-based self-sufficient 
system, which includes automatic image capture and 
real-time object recognition. Besides, we discussed its 
validity in promoting healthy olfaction and pleasantness 
through a brief user study. Finally, we present other 


application possibilities of OlfacEnhancer in the future. 


Keywords 


Olfaction Augmentation, Mobile Object Recognition, 
Cross-Modal Perception, Smell dysfunction, Olfactory 


Interface 


1. Introduction 


Hyposmia, or decreased sensitivity of olfaction, is 
common among both elderly and younger people, which 
has affected their quality of life. It is also a potential 
symptom of Alzheimer’s dementia as well as other 
neurodegenerative diseases [2], as olfaction is intimately 


connected to the brain system and associated with 


emotion and memory [1]. Researchers have proved 

that daily exposure to certain odors would improve 
olfactory perception as well as the condition of 

related diseases [3], while “vision-smell” multisensory 
stimulation in the natural environment could further 
increase pleasantness [4] and enhance the healing 
effects [1]. However, few devices have been developed 
for that in spite of the emerging technology of computer 
vision. 

In this paper, (1) we propose OlfacEnhancer, a wearable 
olfactory interface for hyposmia patients, which 

can release seven different scents synced to visual 
information. It’s a novel application of computer vision. 
(2) And we develop an automatic system that doesn’t 
require any manipulation during use. (3) A brief usability 
test indicated that OlfacEnhancer could create “smell- 
vision” synaesthesia experiences, improve the quality 

of life and olfaction health, which supports many other 


unique application scenarios. 


2. Related Work 


Cross-modal solutions for sensory augmentation have 
been widely discussed. Recently increasing studies focus 
on olfaction enhancement for notification, memory 
recalling, interpersonal communication, emotion 
regulation [10], and contextual interaction [5]. These 
systems evoke or augment the sense of smell linked 


with memory and emotions [1]. However, in a real- 
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world environment, context-based scent delivery, 
with correspondence to other senses, would enhance 
immersion in more practical and promising application 


scenarios. 


The applications of CNN models on embedded vision 
system become feasible since Depthwise Separable 
Convolution and MobileNet [9] were proposed. 

Now mobile object recognition devices have been 
widely developed, some of which create cross-modal 
immersive experiences. To connect vision and olfaction, 
Kim et al. [6] improved CNN method to recognize 
odor-evoked objects from images and emit particular 
scents. However, users are supposed to take photos 
themselves with an Android device before a sniff, which 
may interrupt synaesthesia. We propose a compact 
design integrated camera and multi-scent display to be a 
self-sufficient system, which requires fewer steps and is 


easier to use, especially for the elderly. 


3. Design Description 


To enhance olfaction synced to visual data for daily 
use, OlfacEnhancer includes a camera for image 
capture, scents releasing system and a control and data 


processing center based on Raspberry pi zero. (Fig.1) 


3.1 Multiple Scents Releasing 


To bean individual-scale multi-scent output device, 
OlfacEnhancer has a compact design. A scent is selected 
when the servo-driven turntable makes its only vent 
exactly align to the scented division. Each division 
contains pre-packaged essence and has an air passage 
of stuffed cotton sandwiched between waterproof and 
breathable membranes. Vaporized essence spreads 
through the membrane and is temporally absorbed by 


the cotton until diffusing into the air when the vent is 


opened. 
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During this process, cotton plays an important role to 
control the volatilization speed and scent intensity, as 
olfaction threshold differs with individual and original 
essence. Between turntable and divisions, a film of 
adhesive cotton could prevent cross-contamination with 
minimum resistance to rotation. Besides, to prevent 
olfactory adaptation, there is 40 seconds' interval 
between 3 seconds' odors releasing in succession [7]. 
During the interval, servo rotates and stops at the 
midpoint between two air vents. Therefore, the odor 
substances in the air have dissipated before the next 


release. 


3.2 Automatic Object Recognition 


We made the preliminary prototype to recognize 
seven common scent-evoked plants (the labels in Fig. 
3) and trained a MobileNet model with 8 outputs 

(with a blank output indicating "no flowers in the 
image"). The architecture of the model is in Fig. 2. A 
pre-trained model with ImageNet dataset accelerates 
the convergence. We modify the last fully-connected 
layer of MobileNet and fine-tune it on our own dataset 
with positive samples of 7 scented objects and some 
negative samples. Our model gets an accuracy of 79% 
on test-dataset and is deployed on Raspberry Pi zero 
with Tengine [8]. Fig.3 illustrates the Confusion for the 
performance. In an actual measurement, our system 
identified most of the testing objects no more than 2 
meters straight away. Therefore, we improve the system 
with automatic clipping, and achieve the recognition 

in an extended range of 3-4 meters, with a reduced 


processing speed of 2s interval between 2 shots. 


4. Evaluation 


To evaluate the usability of our preliminary prototype, 
we conducted a brief study to investigate the 


experience with OlfacEnhancer. 
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Fig. 2. (left) The architecture of MobileNet. Input: a resized image with the size of 224x224 and 3 channels(RGB); Conv: the output 


of convolution layer; AvgPool: the output of average pooling layer; FC: the output of fully-connected layer; Softmax: a classification 


layer that output probability of each class. 


Fig. 3. (right) Confusion matrix for the performance of our model. The horizontal axis is the label predicted by MobileNet while 


vertical axis the true label of the test dataset. All data are normalized by the number of true labels of each class. 


We recruited 4 representative subjects (2 males and 

2 females, mean age = 32, sd = 10.49). They were 
required to take walks twice (wear OlfacEnhancer 

and wear nothing) and meet one of the plants every 5 
meters. We evaluated the time it took to produce smell 
perception by recording the human-object distances 
when subjects smell the scent. After each walk, they 
were asked to rate odor intensity, fidelity, degree of 
comfort, immersion and pleasantness in a visual analog 
scale (VAS). 


The outcome of the human-object distances wearing 
OlfacEnhancer is 1.5-3.6 meters, while in the controlled 
trial is no more than 0.5 meter, which indicates that 
wearing OlfacEnhancer evokes smell perception in 

a longer distance than not. In addition, results of 

VAS assessment show that wearing OlfacEnhancer 
comfortably augments olfaction, enhances immersion 
and enjoyment although some kinds of scents are 

not natural enough. According to the experiment by 
Salminen et al. [4], viewing experience with odors was 
always rated as pleasant, and the level is affected by 
odor authenticity. Therefore, we infer that a selection 
and synthesis of high fidelity essences will improve the 


prototype. 


Overall, OlfacEnhancer performs well in augmenting 


olfaction and improving pleasantness and immersion. 


However, we should further test its validity and 
robustness through more rigorous experiments with 


hyposmia elderly as well as dementia patients. 


5. Applications 


OlfacEnhancer is easily reproduced to be a new version 
for many future opportunities. We present some of the 


scenarios in this section. 


5.1 To Create Immersive Experiences 


Immersive exhibition. Wearing OlfacEnhancer in 
the exhibition, users would have strong enjoyment 
and impression the artworks. It supports different art 
forms, such as sculptures, paintings, and videos, and 


various positions in a distance and from any perspective. 


Preserving Cross-Modal Memory. OlfacEnhancer 
could preserve smell profiles corresponding particular 
sight during travels that will be the best souvenir to 


bring back the memory with more emotions loaded. 


Enjoying Allergy Seasons. People suffering from 
hay fever always stay away from floral aromas, which 
may carry spore or other allergenic ingredients. An 
OlfacEnhancer containing healthy alternative essences 
will help them enjoy the aromas in a distance from the 


scented plants with reassurance during allergy seasons. 
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5.2 To Evoke Vision-Related Feelings 


Building Up Expectations. By giving pleasant scents, 
OlfacEnhancer could trigger the appetite and curiosity 
of children, and thus the willingness to explore new 


food, books or other things in front of them. 


Empathizing. If combining expression recognition 
method, OlfacEnhancer could produce the scents 
semantic to the emotions of the person in front. That 
could evoke the same feelings with other people and 


promote interpersonal understanding. 


Scentification. Certain scents could trigger attentive 
and conscientious emotions. Therefore, OlfacEnhancer 
could provide real-time alerts and general arousal of 


attention before some dangerous manipulations. 


6. Conclusion and Future Work 


In this paper, we present the design guideline and 
implementation of OlfacEnhancer, a vision-based multi- 
scent pendant. Our formative tests show that it indeed 
enhances immersive perception and pleasantness via 
releasing fragrance when the scented-object comes 
into view. Overall, we believe our work is a creative 
exploration. In future work, we will perform further 
studies to assess the olfactory augmentation effect 

on multiple user groups with the improvement of the 
odor fidelity as well as the processing speed. Further, 
we hope the proposal of OlfacEnhancer could remind 
readers of the important role of smell and paying 


attention to olfaction health. 
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APOSEMA: Exploring Communication 


in an Apathetic Future 


Abstract 


Aposema is a wearable device that speculates on a near- 
future scenario in which human face perception abilities 
have been compromised. In this world, everyone wears 
a device on their face that reads facial expressions and 
guides social interaction. The device is composed of 
three elements: The first is the sensory input from 

the face muscles, followed by a data analysis. The 
second is the physical output—a soft, colored, robotic 
inflation pattern. The colors represent the emotions 
corresponding to the facial expression, and the form 
change represents the emotional intensity. The third is 
an augmented reality layer that provides a decrypted 
emotional analysis to help guide interaction. Aposema 
is a personally customized device made of silicone 

and incorporated electronics. The fabrication process 
involves parametric design, 3D printing, hand casting, 
and physical computation. This design is proposed as 

a critical object and a provocation responding to the 


changing nature of communication in the digital age. 


Keywords 


Wearable Design, Speculative Design, Face Perception, 


Soft Robotics, Augmented Reality 


1. Introduction 


Aposema is a wearable device that is worn on the 

face (Figure 1). It is proposed as a piece of speculative 
design and a physical, critical object positioned in the 
intersection of art, science fiction, and product design 
[11], [16]. Its goal is to provoke thought and raise 
discussion regarding the challenges and opportunities 
for communication that our society may soon encounter 
as wearable technology evolves in the near future [12]. 
Motivated by the digital revolution, the information 
technology age, and more recent developments in the 
wearable technology industry, Aposema explores an 
imagined oncoming crisis and the increasingly extreme 
responses we might take to mitigate it. This scenario 
draws from our extensive use of personal digital devices 
and social media. As part of the digital revolution, our 
communication habits have changed, with individuals 
increasingly choosing technological alternatives over 


unmediated in-person interaction [2]. 


2. Narrative as Design Compass 


We imagined a scenario inspired by last century’s 
science fiction cinema and literature to guide the design. 
Aposema speculates on a near future where we rely 

on technology to replace our once-natural instincts. In 
an age of genome engineering and emotion recognition 


algorithms, our ability to read facial expressions has 
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been severely reduced, limiting our capacity to develop 
relationships and leaving us struggling to empathize. 

In this world, we attempt to compensate for this 
impairment with a wearable technology device. 

As we developed our narrative, we explored several 
research areas of wearable devices for guidance. Our 
starting point was previous work that used soft robotic 
wearable devices as instruments for speculation on 
future scenarios, such as Ava Aghakouchak's and Maria 
Paneta's Sarotis [1]. In addition, we looked at previous 
work that used wearable devices to guide social 
interaction, for example Behnaz Farahi's Caress of 

the Gaze [6]. Finally, we based our design on previous 
work that explored facial expression recognition 
through wearable devices, such as Jocelyn Scheirer's, 
Raul Fernandez's and Rosalind W. Picard's Expression 
Glasses [7]. 


3. The Three Parts of Aposema 


Aposema is proposed as a body extension providing 
its wearer with an emotional and cognitive extension 
[3], [17]. It is a device that reads facial expressions and 
analyzes emotional states to guide social interaction. 
It is composed of three parts (Figure 2): The first is 
the sensing part or the input. A system inside the 
device, composed of a programmed microcontroller 
and biometric sensors, reads the facial expression and 
integrates it with other sources of personal data about 
the user (Figure 3). The device then creates an analysis 
aimed at replacing the natural complex process of 


natural face perception [4]. The second part is the 


presentation of information on the facade of the device. 


The encrypted analyzed information is presented as 


a soft, colored, robotic pattern. When the person 


Expressions Sensing & 
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Soft Robotic 
Encrypted Information 


Fig. 1. One ofthe final design iterations of Aposema (Source: 
Aposema, design by Adi Meyer, Sirou Peng, and Silvia Rueda, the 
Interactive Architecture Lab, the Bartlett School of Architecture, 


University College London, 2017) 


wearing the device encounters other people, the third 
part—the decryption—comes into action: the device 
decrypts the soft robotic patterns on devices worn 

by others. An overlay of augmented reality viewed 
through a lens incorporated within the device provides 
the wearer with a decrypted analysis of other devices 
around them, which enables and enhances interpersonal 
understanding (Figure 4). The three elements of the 
device form a system meant to compensate for the loss 


of the ability to read facial expressions (Figure 5). 


4. Replacing Face Perception for Social 
Interaction 


The device aims to replace the complex process of 

natural face perception through simplification. Face 

perception is, at its root, an individual's understanding 

and interpretation of the human face when encountered 

as a visual stimulus. The ability to process information 

Fig. 2. The device is 
composed of three 
main parts. (Source: 


Aposema, 2017) 


Augmented Reality 
Decrypted Information 


Fig. 3. (Left) Sensors 
integrated within the 


device read facial 


muscle movements and 


analyze them. (Source: 


Aposema, 2017) 


Fig. 4. (Right) An 
overlay of augmented 
reality provides a 
decrypted analysis of 


other devices. (Source: 


Aposema, 2017) 


AR Decrypted Information 


from faces by recognizing facial expressions involves 
extensive and diverse areas of the brain [4], [8- 

10]. The sensors in the device read facial muscle 
movements, process them, and translate them into 
simplified emotional states according to the universal 
emotions identified by psychologist Paul Ekman. 
Ekman ascertained six universal core emotions—that 
is, emotions that are perceived and expressed in a 
similar manner in every human culture. These emotions 
are disgust, sadness, happiness, fear, anger, and 
surprise. Through his research, Ekman classified facial 
expressions by motion cues corresponding to these 


distinct emotions [5]. 


After the wearer's expression is read and interpreted, 
an emotional analysis is presented on the device. 

The facade of the device is made of silicone and has a 
colored pattern with inflating pockets of fluid. It reacts 


when the wearer interacts with another individual, 
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Fig. 5. The elements of Aposema (Source: Aposema, 2017) 


representing emotion by changing color and shape. 


Each color that appears on the device represents one 
of Ekman’s universal emotions, and the degree of shape 


change indicates the intensity of feeling (Figure 6). 


The device constantly scans the wearer’s environment, 
recognizes soft robotic patterns presented on other 
devices, and decrypts them in real time (assuming that, 
in this future world, we are all wearing those devices). 
The goal of the lens over the eye area of the device is 
to provide an augmented reality overlay that equips the 
wearer with further interpretation and an emotional 
analysis of encountered individuals (Figure 7). Alongside 
the analysis, a prescribed guideline for social interaction 
is provided. When the device recognizes an expression 
of anger, it might suggest pacifying hand gestures; 

when it recognizes a sad expression, it might suggest 
comforting words. Through this two-way transfer 


of information, the device facilitates a new form of 


expressive communication. 


Fig. 6. Each color presented on the device corresponds to one 


of Ekman's universal emotions. (Source: Aposema, 2017) 
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5. The Technical Process of Fabricating 
Aposema 


The technical process of producing Aposema began 
with creating a three-dimensional scan of the wearer 
for optimal personal customization. Once the scan 
was produced, the device was digitally modeled using 
parametric tools to guide the pattern design. The 
digital model was then translated into a physical object 
by 3D printing a series of molds in which the device 
would be casted. In parallel, a circuit was designed, 
and a microcontroller was programmed to sense facial 
muscles and translate them into physical and digital 
outputs: the soft robotic inflation and the augmented 


reality layer (Figure 8). 


6. Learning from the Past to Imagine the 
Future 


Aposema derived its name from nature's aposematic 
visual warning system and from mask-making traditions. 
Aposema's notion of dynamic representation is based 
on Kwakiutl dynamic tribal masks from the 18th- 

19th centuries. It interprets the concept of dynamic 
representation through a physical object by utilizing 
advanced technologies for emotional representation in a 
device that, like the masks, physically transforms and is 
worn on the face. The Kwakiutl Indians of North British 
Columbia produced spectacularly designed masks that 
were a constant feature of their rituals. Their masks 
were viewed as a way to change the wearer's identity; 
they contained a dynamic transforming element, as 

they could be opened to reveal an inner layer [13-14]. 
The dynamic quality of these masks was an important 
precedent for Aposema. The Kwakiutl masks helped us 
imagine an abstraction of current wearable technology 
to throw a spotlight on our changing relationships with 


the virtual and physical worlds. 


Although the speculative scenario that drives Aposema 
is grim, in a sense, it resembles our current reality. It is 
not to be mistaken for a technophobic dark prophecy. 
It is merely a provocative narrative aimed at addressing 
the issues accompanying the rapid technological 
advances that our society has been experiencing in 

the decades since the digital revolution. While we 
acknowledge the contribution of digital devices to 


our well-being, the way that individuals consume 
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- 


Fig. 7. The soft robotic pattern is read through an augmented reality overlay, 


equipping the wearer with further interpretation and an emotional analysis of 


encountered individuals. (Source: Aposema, 2017) 


information has deeply changed, and we as a society 


hold the obligation of overseeing the impacts of this 


transformation. We must be aware of the extent to 


which our communication paths are managed, directed, 


and manipulated so that we can assess the challenges 


and opportunities presented by future developments in 


wearable technology. 


7. Conclusion 


We are more connected than ever, yet our connections 


rely on brief, rapi 


d exchanges of information. 


Connections in the near future will be shaped in 


unpredictable ways. This development may have 


consequences that we do not yet have the ability to 


comprehend. As a piece of speculative design, Aposema 


encourages further research on the influence of digital 
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devices and wearable technology on our interpersonal 
connections, as well as the ethics of wearable design 
related to possible violations of privacy as the device 
collects personal data. From increasing surveillance 
to the rapid growth of intelligent wearables, the 
human experience of our immediate environment is 
changing. These technologies combined with social 
media play a pivotal role in shaping our relationships 
[15]. The conclusions of this research should be taken 
into account when designing products that have deep 


implications for our social and psychological well-being. 
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HuValue: A Toolkit to Facilitate Considering 
Various Human Values in a Design Process 


Abstract 


Human values play an integral role in any design that 
aims to improve the quality of human life. To support 
designers’ addressing human values in their design, 
we created a design tool based on a comprehensive 
value framework. This tool helps to raise designers’ 
awareness about human values and provide tangible 
materials to actively use selected values in a design 
process. An experimental study with design students 
showed that the project groups that had been 
supported with this tool addressed human values in 
their design concepts significantly stronger compared 
to the control groups. Results of the evaluation indicate 
that the tool is not only applicable in a design process 
but also effective at enriching design concepts with 


human values. 


Keywords 


Design Tool, Human Values, Value Framework, Design 


Process 


1. Introduction 


Nowadays, life without technology is hardly imaginable. 
Technology is interwoven with all aspects of life, and 
people perceive the world via artifacts [1, p. 235]. As 


artifacts are not self-formed phenomena, emphasizing 
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on the role of these products in human’s life brings 

the responsibility of designers in the light [1, p. 234]. 
Nevertheless, while design in its origin “aims at changing 
existing situations into preferred ones” [2, p. 111], 
there are very few agreements on what a preferred or 
better situation means and how it could be achieved. 

In this respect, human values can be introduced as a 
touchstone. In fact, values can be embedded in artifacts 
[3], and values expressed by the designer should be 
interpreted by the user [4]. We use these points to 
argue that considering human values behind every 
human action, goal and decision can provide a wide and 
sensitive view. This view has implications for design as 
the designer needs to understand what is preferable for 
the end-users and to make appropriate decisions [4]. 
Despite the significance of human values in everyday 
life and consequently, in design [5], they mostly remain 
implicit and unarticulated in design projects [6]. Only 
few design approaches concentrate on human values 
and aim to address them in their design: Value Sensitive 
Design (VSD) [7], Value-led Participatory Design (VPD) 
[8], and Value-Centred Design (VCD) [9]. Nonetheless, 
there is very little agreement between them to identify 
values. In this respect, the lack of an established 

and accepted fundamental grounding [10] and a 
comprehensive list of values [11] can be considered as 
a major unresolved issue. Accordingly, our research 
intended to support product designers considering 


human values consciously and explicitly in the design 


Fig. 1. Value wheel 
(Size: A2), with nine 
value clusters and five- 


point Likert-type scales. 


process. The core assumption in our study is that a wide 
view on human values and facilitating using this concept 
in design would help designers to take human values 


into account. 


2. Exploration of a Design Tool Based on a 
Value Framework 


Since the term 'value' is widely used for different 
purposes in various disciplines, and there is no 
comprehensive value framework for design, we 
considered the need for supporting product designers 
regarding human values: a general list of human 
values to cover diverse views as well as a well- 
classified framework to summarize the list in a brief, 
understandable and applicable model and simplify 
thinking about and discussing them. A holistic view 

of the values of different aspects of human life [12] is 
important to improve the quality of humans’ life, and 
this improvement is related to progress in all aspects 
of life [13]. Due to the natural distance between the 
abstract level of human values and practical level of 
design, we considered the need for a tangible medium 


to bridge this gap and facilitate using a comprehensive 


value framework. 


The HuValue tool is designed in the form of a card- 
based design tool (Fig. 1 and Figure 2). The form of a 
card-based tool was selected for the design tool since 
cards are low-tech, tangible, and approachable design 
materials which are an effective medium to bridge the 
gap between theories and practice [14]. The HuValue 
tool contains a value wheel (Figure 1), 45 value words, 
and 207 picture cards (Figure 2). This tool is grounded 
on a comprehensive value framework for design [15]. 
This framework was created and developed via various 
theoretical, empirical and design-based approaches to 
compile, classify and structure the existing value lists, 
including Rokeach [16], Schwartz [17], Peterson and 
Seligman [18], and ten more value lists from the last 
century [15]. In the HuValue tool, the value framework 
is illustrated in a circular structure as the value wheel 
(Figure 1). 


The Value wheel is a circle with nine value clusters, 
each with an icon, a label, a mood board, a descriptive 
sentence, five key values, and some relevant terms. 

This wheel provides an opportunity to express the 
importance of different human values, which can vary 
from person to person and situation to situation. 

The value clusters can be ranked in their order of 
importance via a 5-point Likert scale: 'Extremely 
important’, ‘Very important’, ‘Important’, ‘Somewhat 
important’, and ‘Not important’. The nine value clusters 
are 'Carefulness', ‘Justice’, ‘Ecology’, ‘Respect for 
others’, ‘Meaningfulness’, ‘Status’, ‘Pleasure’, ‘Respect 
for oneself’ and ‘Personal development’. The outer 
circle indicates the relation of the value clusters to four 
general themes, including ‘Basic believes’, ‘Nature’, ‘Self’, 


and ‘Society’. 


The Value words are 45 two-sided cards (5 x 7 cm) 
with a value word on the front and its relevant value 
cluster at the back [15]. These cards represent the key 
values mentioned in the value wheel to provide more 


flexibility for using and emphasizing individual values. 


The Picture cards contain 207 cards (7 x 10 cm) of 
three different types: 66 activities, 66 personae, and 
75 products/services. Picture cards, as complementary 
to value wheel, are examples of activities, personas, 


and products/services that are presented in the format 
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Wisdom 


a mature understanding of life 


Personal 
Development 


in Mex erc 
ue Words | 


Albert 
« Einstein 


Einstein (1879 — 1955) is acclaimed as one of the. 


August Smart Lock 
Yves Behar 


A keyless lock system that hooks onto your 
‘existing dead bolt and lets you unlock doors 
through mobile applications. Far homeowners, the. 
hardware will enable easy access to a house for 


equivalence formula E = mc2 (which has been dubbed emnsalvos an wal an COL Hun MER 


most famous equation’) 


4 "m "nm 
cture L 


Fig. 2. Examples of value words and picture cards: (Left) two-sided cards with a value word on the front and its relevant value 


cluster at the back; (Right) three different types of picture cards including activity, persona, and product/service. 


of cards. Despite the simplicity of their presentation, 
these cards' role is to link the abstract human values 

to everyday life. In fact, the cards are supposed to 

be applied for expressing human values in practice; 

the activity cards are some examples of the possible 
valuable behaviors, which can be used to express what 
does a specific value/group of values mean and how 
does it appear in a real life; The persona cards are the 
examples of iconic people, who can be representative 
of acting based on a specific value/group of values in 
life; and the products/services cards are some examples 
that can be used to try expressing how using a product/ 
service in daily life can straighten or weaken a value/ 


group of values. 


Generally, the HuValue tool is a mean to facilitate 
thinking about and discussing human values. This tool 
supports designers with simple but familiar materials 
during their design process to analyze everything 
(object/subject/situation) from a wide value point 

of view. This approach, referred to as the HuValue 
perspective, enables the designers to be aware of and 
sensitive to human values and consider various aspects 


of their topic and different types of values. 


Being aware of the diversity in design processes, 

the HuValue tool is intended to be used for several 
applications based on common design activities such as 
analyzing the situation, defining design goal, generating 
ideas, selecting a final idea, developing the concept, 
realizing and evaluating the final concept. In this respect, 
the tool can be applied for various purposes: the 


design challenge, the design goal, the context of use, 
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and the user's needs, desires and requirements can be 
studied from a value-centered point of view for a better 
understanding of the design situation; human values can 
be used not only as a source of inspiration to diverge 
the ideas but also so that they cluster and converge; 
values can be seen as criteria for deciding on the final 
idea; the final concepts can be evaluated from a value 


perspective. 


3. Evaluation 


To investigate the value of the HuValue tool, a quasi- 
experimental study was realized to test the applicability 
and effectiveness of the tool in a design process [19]. 

A group of bachelor students (N = 64, out of 192 
students) were randomly selected to take part in this 
study. After training them about the intended usage of 
the tool, the students were supposed to use the tool 

in their design projects. The data were collected via 
questionnaires and the students' final deliverables of 
their conceptual design. The outcomes showed that 
the project groups who were supported with this 

tool addressed significantly stronger human values in 
their design concepts compared to the control groups. 
Results of the evaluation indicated that the tool is not 
only applicable in a design process but is also effective at 


enriching design concepts with human values. 


4. Conclusion 


Considering the outcomes of our study, we can 
conclude that raising awareness about human values 


and facilitating using a value point of view seems to be 


helpful to emphasize human values in design. However, 


the current study is limited by time, and further 


investigations and developments should be realized. 


Next steps are to digitalize the tool and make it more 


simple and persuasive to use. We also recommend 


investigating how to apply the tool for changing behavior 


and how to use it for preserving and/or changing values. 
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Playing with Systems: Tactile Games as 


System Prototypes 


Abstract 


The world’s most pressing challenges are systemic. 

Our ability to survive as a species is now linked to 

how quickly and effectively we are able to address 
systemic challenges like climate change. The practice 

of low-resolution prototyping helps designers quickly 
explore and learn about ideas, but it is rarely utilized 
when designing for complex systems. At the same time, 
physical games have often been utilized in service of 
learning about complex systems, but they have not 
been used as prototypes for systemic interventions. 
This short paper and demo provide examples of two 
games that show how designers can use games as 
low-resolution prototypes for complex systems. These 
physical games will transform systems from intimidating, 
impenetrable, and abstract concepts, to entities that can 


be explored, engaged, and influenced. 


Keywords 


Climate Change, Prototyping Systems, Design, Games, 
Play 


1. Introduction 


Climate change is a systemic challenge that poses an 


unprecedented threat to life as we know it. Responses 
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to climate change will require the deployment of an 
unprecedented breadth of initiatives at a global scale 
in little over a decade.[1] In order to be successful, we 
must be able to rapidly explore ambitious ideas as well 
as the risks they entail. Low-resolution prototypes are 
well suited to this task and although they are widely 
used at the product and service scale, they have yet 

to be employed in the design of systems. This paper 
demonstrates the ways physical games can be used 

as a low-resolution prototypes of a systems-level 
interventions and show how employing this type of 
practice can help experts and non-experts engage with 


complex systems in a tangible way. 


2. Complex Systems 


While complex systems are varied in form and scale, 
for the purposes of this paper they will be categorized 
by the work of Yaneer Bar-Yam. According to Bar- 
Yam, these types of systems share the common traits 
of a diverse range of functions, interconnections, 
interwoven elements that each relate, and influence 
one another.[2] Using this framework, the challenge of 
climate change is one of our best examples of a complex 
system. Its causes are layered and inter-related and any 
meaningful responses to it will require shifts in the way 
multiple systems (such as national, regional and local 


governments) behave. 


3. Prototypes as Tools for Learning 


This work builds upon the notion of prototyping from a 
design thinking perspective. In this domain, the practice 
of prototyping can be described as the rapid creation 
of artifacts to learn about a solution space. Prototyping 
is widely understood in the product and service design 
realm as a way to explore, gain understanding, and 
reduce risk.[3-7] Within this practice, prototypes 

are broken into low, medium and high fidelity. Lower 
fidelity, or low-resolution (low-res) prototypes are 
much faster to build and are easier to make, but they 
bear little resemblance to the final version of an idea. 
[8] When creating low-resolution prototypes, designers 
focus their efforts on building only what is needed to 
elicit feedback about critical areas of the design from 
their users. In this way, low-res prototypes allow 
designers to quickly learn and iterate towards more 


refined solutions. 


4. Educational Games 


Although the use of games in service of learning is 

not new, the work of Dieleman and Huisingh (2006) is 
instructive for the discussion of prototyping because 
they highlight the connections between games and 
experiential learning.[9] Some of the benefits Dieleman 
and Huisingh attribute to playing games, specifically 
the focus on experimentation and failing without real 
consequences, could easily be used to describe the 


benefits of low-res prototyping. 


Learning games and prototypes do share many common 
traits, but there is an important distinction between the 
two. Educational games use play in service of a learning 
objective. While games-as-prototypes certainly offer 
some educational benefit to the player, their primary 
purpose is to help the design team understand as much 
as possible about the design space in the least amount 


of time. 


4.1 Examples of Low-Resolution Systems 


Prototypes 


In the following games, a complex system is translated 
into a physical experiences and objects that help a 
variety of stakeholders engage with these systems in a 
more meaningful way. These games reflect a humility 


that acknowledges the limitations of our ability to fully 


understand or account for all of the potential forces 
and variables within a system. Like any low-resolution 
prototype, these games do not attempt to replicate 
the full complexity of a system. Instead, the games 

are designed with just enough context and game 
mechanics to spark visceral experiences of system level 
changes. The simple quality of the game serves multiple 
purposes. It allows the games to be played by a wide 
range of users and it minimizes the effort required to 
create the prototype/game itself. If the game becomes 
too complicated, the focus will shift from the larger goal 


and intervention to the prototype. 


"Earth Systems and Modern Convenience" serves as a 
prototype for a hypothetical initiative intended to 
promote sustainable practices. It utilizes the relationship 
between gravity and distance from the floor to simulate 
the ways our planet's natural and human-systems 
become more fragile as our population grows. The 
game, grows more challenging at the later phases. As 

a prototype, the game will be successful if it can help 
the designers gain greater confidence in a proposed the 
direction before committing any more time or money 


than is necessary. 


The game has two key parts, a wooden disc balancing on 
a thin column and a series of small wooden blocks. The 
24" wooden disc represents the Earth. The height of 
the column on which it rests corresponds to the Earth's 
historical (or projected) populations. The wooden 
blocks each represent mundane elements of modern life 
such as owning a car, eating non-local food, or utilizing 
same-day delivery services. The object of the game is to 
place as many blocks onto the disc without causing it to 
topple. Through the course of the game, the players are 
forced to make difficult decisions which reveal a variety 
of strategies, connections and intentionality about 
choices that are often unnoticed in daily life. The tactile 
nature of the game leverages existing systems, (gravity, 
the player's nervous system and the uncertainty of the 
immediate environments) which all become inputs for 
the game. As an example, a heavy footed pedestrian 


could throw off a precariously balanced disc. 
If the debrief is a critical part of an educational game, 


as noted by Dieleman and Huisingh, it is absolutely 


essential for a prototype.[10] The debrief offers the 
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designer a chance to understand more about the 
player's choices and experiences during the game. The 
phase of the design process, commonly referred to as 
"testing", helps designers understand more about their 
users and their proposed solution. At the Stanford 
University d.school, the process of testing is often 
referred to as “empathy with a prop". [11] The debrief 
after a game serves a similar purpose and the insights 
gained in this phase help inform the next iteration of a 


solution. 


During the debrief of the game, it is critical that the 
designer remains open to the variety of discussions 
that emerge, even if they challenge the premise of 

the game. For example, in "Earth Systems and Modern 
Convenience," players may rightfully question why all the 
blocks are weighted the same when the effects of single 
day delivery and non-local food may have very different 
impacts. In remaining open to these kinds of discussions, 
the designers are able to get a better understanding 

of the ways which the players view the system and 

this creates space for the players to become design 


collaborators as well. 


"Critical Responses to Climate Change" comes out of 
the OneClimate initiative at University of California, 
Davis which is spearheaded by Dr. Benjamin Houlton, 


Director and the author who is on the leadership team 


A player tries to fix a leak 
with a carbon neutral solution 


Magnets provide 
player feedback 


for the initiative. In order to break down the complexity 
of climate change into an actionable (although highly 
ambitious) plan for our species' survival, we distilled the 
challenge into three key activities. As a planet, we must 
remove carbon from the atmosphere, change human 
behavior to adopt carbon neutral lifestyles, and do all of 
this while adapting to the effects of climate change that 
are already underway. In order to share this approach 
with a wide audience, the author developed a metaphor 
in which the Earth is a leaking boat. To survive we 

need to bail the boat and fix the leak in the growing 
turbulence caused by climate change. An animated video 
describing the approach can be seen here. (https:// 
vimeo.com/270732552) 


This game translates the metaphor of a sinking boat into 
a tangible format that is suitable for elementary school 
and above (Fig. 1). The leaks in the boat represent 
carbon accumulation in the atmosphere. The boat will 
sink unless the players are able to bail the boat (pulling 
CO2 out of the atmosphere), fix the leak (changing 
human behavior in order to adopt carbon neutral 
practices) and shore up the boat (responses to the 
effects of climate change that are already underway). 
This game translates some of the climate jargon and 
abstract concepts into physical experiences that help 


players experience climate change in a visceral way. 


A player tries to bail the boat 
using a tool sized to represent 
afforestation techniques 


A "leak" representing 
r current carbon emissions 


~ 


“People” representing current 
carbon consumption behaviors 


€— — — — — ——- Temporary walls (representing climate-smart 
infrastructure) provide some modest defense 
against the incoming water. 
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Fig. 1. An image of the 
game in action with 
key elements of the 


prototype called out 


Players may work as an individual or in teams. They are 
not allowed to touch the boat and they have a small 
amount of time (six minutes) to keep the boat afloat. 
The leaks in the boat correspond to the proportions of 
carbon emissions and the bailers are sized to represent 
the latest figures on carbon sequestration. As with Earth 
Systems and Modern Convenience, the efficacy this game 
is deeply tied to the debrief and facilitation after the 
game play. In the game, the role of water, magnets and 
collaborative efforts between players all become rich 


sources of insight. 


5. Conclusions 


In this paper, the author provides a theoretical 
framework for the ways which games can be used as 
prototypes for interventions in complex systems. These 
low-resolution prototypes can provide insight to both 
players and game designers which provides the ability 
for designers to gain greater insight into potential 
system interventions in less time and with less risk. The 
games themselves have been designed and prototyped, 
but more follow-up research is needed to explore the 
potential and shortcomings to this approach. As tools 
for learning they are not intended to replace the full 
complexity of a system, but instead offer playful ways 
of quickly eliciting information about the dynamics, 
relationships and unexpected influences in the system 


for both player and designer. 
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Attributes of Aliveness: 
A Case Study of Two Interactive Public Art 


Installations 


Abstract 


What elements make people think that an interactive 
public art installation seems “alive”? To answer this 
question, the paper examines the design and the 
reception of two public artworks by the INVIVIA 
studio: MIMMI and PULSUS. Both are responsive 
installations that aggregate collective data and interpret 
it through elements such as vibrations, lighting, and/ 

or mists, but differences in design decisions have led 

to different perceptions of their aliveness. Through 

the case study, this paper identifies key attributes in 

an installation that contribute to the sense of aliveness 
and discusses the importance of considering aliveness in 
design. While these attributes are distilled from public 
art examples, they can generally apply to designed 
objects that are, through technological augmentation, 


physically interactive at the human scale. 


Keywords 


Aliveness, Perception, Human-Technology Interaction, 


Technological Augmentation, Public Art Installation 


1. Introduction 


What makes an interactive public art installation seem 
"alive"? The question of perception of aliveness has 


been explored by many interaction researchers. The 
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criteria for perceived aliveness has changed over the 
years, shifting from autonomous motion, to independent 
intelligence, to possession of the capacity for reciprocal 
connection [1] and the ability to share control with the 


human actor as a co-agent in negotiation [2]. 


In this research context, this paper presents a case 
study of two public art installations to investigate 

how design decisions led to different perceived levels 
of aliveness as experienced by visitors. Through this 
case study, the paper identifies key attributes in an 
installation that contribute to the sense of aliveness and 
discusses the importance of considering aliveness in 


design. 


2. Case Study of Two Installations 


To investigate what qualities might contribute to the 
aliveness of interactive public art installations and how 
people respond to these qualities, this paper presents 
two public art installations - MIMMI and PULSUS 

— designed by the INVIVIA studio and on which the 


authors have collaborated. 


2.1 MIMMI 


Overview. The first case was MIMMI [fig. 1], the 
winning entry to the 2013 Creative City Challenge Art in 
the Plaza competition in Minneapolis [3]. It was a large 


air-pressurized sculpture hovering over the Convention 


Wes 
al Lr Ke 


Fig. 1. Installation view of MIMMI. 


Center Plaza. It analyzed real-time emotive information 
gathered via Twitter from city residents to create 


changing light displays and mists in response. 


Design. MIMMI was designed to be a round, light, 
cloud-like presence in the middle of the plaza. The 
installation responded to moods of the city with 
colored lights and mists. The core algorithm aggregated 
real-time geotagged tweets from the city, evaluated 
their emotive state, and translated that state into a 


dynamic microclimate under the suspended structure. 


Responses. The experience was described by visitors 
as "immersive" and evocative of "wonder." Some 
compared the installation to “a magical cloud" to 
comment on its real-time responsiveness. Given the 
narrative of translating the emotional state of the city, 
a group of visitors held a yoga session underneath the 
structure in hope of bringing calming change to the 
installation and taking advantage of the (literal) mood 
lighting. The overall sentiment was that MIMMI was a 
thing that had become “enchanted,” as if its qualities 
of wonder were given to it rather than them being 
intrinsic. Descriptors specifically related to aliveness 


were not mentioned. 


2.2 PULSUS 


Overview. The second case was PULSUS [fig. 2], an 
interactive installation commissioned as part of the 
2017 SummerStreets Festival in Manhattan [4]. The 

large concrete sculpture acted as both a bench and an 
urban instrument that reflected the activity level of its 
surroundings. It gathered real-time digital activity of the 
city and reinterpreted it into an immersive experience 


with vibrations, mists, and sounds. It was installed in 


Fig. 2. Installation view of PULSUS (in NYC). 


NYC for three weeks before relocating to the Harvard 


Graduate School of Design for six months. 


Design. PULSUS consisted of a series of four sections 
with each piece made through a fabric-form process 
with mannequins, resulting in undulations that suggested 
human forms both above and beneath the drapes. The 
sculpture composed a dynamic soundscape based on 
the digital activity of the city, and the soundscape was 
transmitted via sound transducers through the concrete 
as vibrations that were also audible through bone 
conduction. Through an array of piezo microphones, 
the sculpture was locally responsive to taps on the 
surface, eliciting mists from embedded nozzles. The 
intention was to activate the concrete as an interface 
and engineer surprises into the act of interacting with 
an architectural material commonly regarded as weighty 


and inert. 


Responses. The design team initially expressed 
concerns about how people might find the human 
figures in the concrete macabre, but many visitors 
actively interacted with the sculpture as if it were 
something very much alive. Children playing on it 
were seen with their ears and bodies pressed against 
the folds to feel the vibrations of a "lumbering 
creature." Patting the concrete to trigger the mist, 
several remarked that it seemed like the sculpture was 
"breathing." Many visitors were delighted to discover 
sounds akin to "snoring" when they listened with their 
heads against the sculpture. Some even lay down and 
snuggled with the human-like forms. The team did not 
expected the extent to which those who interacted 
with the installation described the sculpture with words 


associated with aliveness. 
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2.3 Different Levels of Aliveness of MIMMI and 
PULSUS 


At first glance, the forms of the two installations 
immediately set up different expectations. PULSUS 
incorporated humanoid forms in its concrete folds and 
invited expectations that these forms might "breathe" 
or "snore." The toroidal shapes of MIMMI did not 
resemble common life forms, so visitors tended not to 


hold similar expectations of aliveness. 


The two installations also offered different potentials 
for relationship development between the visitor and 
the sculpture. While both installations aggregated and 
reinterpreted collective data, PULSUS also offered 
personal intimacy by inviting visitors to physically sit or 
lie in the concrete nooks. The visitor and the sculpture 
could form a one-to-one connection through physical 
contact, and in return the sculpture offered what Turkle 
coined as the "fantasy of reciprocation" [1] through 
vibrations and murmurs. MIMMI did not offer similar 
opportunities for one-to-one connections; the visitor 
had to interact with MIMMI as one person among 

a crowd, and there were no possibilities of physical 


contact between the visitor and the sculpture. 


The interaction mechanisms were not immediately 
decipherable for either installation, and this mystery 
contributed to the sense that the sculpture might have 
minds of their own. To a degree, people understood 
how they might elicit a response: MIMMI required a 
tweet, and PULSUS read social media and listened to 
taps on its surface. But the response pattern was never 
entirely transparent. For both pieces, the algorithms 
aggregated and reinterpreted the data with sufficient 
complexity that the lights and sounds produced were 
more expressive than translational. The PULSUS tap- 
and-mist mechanism also had an element of randomness 
in its response delay and duration, rewarding visitors 
with a level of consistency without becoming entirely 


predictable. 


The different technologies employed in the two 
installations engaged visitors differently depending 

on their familiarity with the mechanisms. The light 
show that illuminated MIMMI was essentially a light 
projection, a form of technology familiar to the present- 


day onlooker. In contrast, the sound transduction 
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technology used in PULSUS was less widely known, and 
this lack of familiarity contributed to a stronger sense of 


wonder. 


3. Attributes of Aliveness 


The interaction insights are distilled into a set of four 
attributes that describe the capacity for an installation 


to be perceived as being alive. 


1. Superficial Resemblance. At a superficial level, 

an installation that exhibits formal or gestural 
characteristics resembling those of known lifeforms 
invites expectations and perceptions of it being alive. 

2. Relational Intimacy. An installation is more likely 

to seem alive if it offers the capacity for people to 
interact with it intimately — at the human scale if it is 
architectural, and in a personal manner if the context is 
public — and if it suggests reciprocation of that intimacy 
through its own feedback system. 

3. Ambiguous Predictability. Human-technology 
interactivity should be neither entirely predictable nor 
entirely random. Ideally, the interaction suggests a logic 
that entices a person to follow along but maintains some 
element of chance to keep the person intrigued. 

4. Technological Familiarity. The quality of aliveness is 
highly subjective, and this subjectivity depends on 

the observer's familiarity with technology. Whether 

a mechanism is "sufficiently advanced" to be 
"indistinguishable from magic," as Arthur C. Clarke 

has put it, depends on prior experience of the person 


making the distinction. 


While these attributes are drawn from two interactive 
public art precedents, they can generally apply to 
designed objects that are, through technological 
augmentation, physically interactive at the human scale. 
It is important to note that the discussion focuses on 
aliveness as an embodied, tangible quality not derived 
from an external source that employs the object as a 


software agent. 


4. Conclusion 


Over the past two decades, there has been a steady 
trend toward the development of technologically 
enhanced things that present themselves as having 


"states of mind that are affected by their interactions 


with human beings" [1]. As technology in computational 
systems and artificial intelligence advances, humans will 
come into more frequent contact with augmentation, 
and human-technology interaction will become 
increasingly complex. Thus, the quality of aliveness is 
important not only because aliveness brings playfulness, 
a quality attractive to humans, but also because 
understanding what makes something seem alive will 
help people build stronger rapport with objects or 
installations. Designers can use these attributes of 
aliveness as first-steps in considering their role and 
responsibility in fostering and mediating this potential 


relationship. 
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Understanding User Customization Needs: 
Requirements for an Augmented Reality 
Lamp Customization Tool 


Abstract 


Nowadays, people are willing to pay for personalized 


items that satisfy their preferences and distinguish them. 


Previous work has provided generic customization tool 
design guidelines. User requirements were gathered for 
the design of an Augmented Reality (AR) application 
for lamp customization in context. These are required 
to define a product configurator that allows users to 
meet their specific needs. The results of three user 
studies show that customers’ needs are preference fit, 
inspiration and help; freedom and support during the 


customization process; and trustworthy visualizations. 


Keywords 


User Research, User Needs, Product Customization 


1. Introduction 


Small-scale, decentralized and personal production 
processes are becoming highly valued. Willingness 

to pay is higher for self-designed products than for 
standard products. Customers perceive the value of 
self-design products as higher when they meet their 
preference system (preference fit) [1]. To ensure the 
delivery of personalized products with high preference 
fit, users must have access to customization tools. 


Albeit rich, previous work’s guidelines for customization 
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are generic and insufficient to design customization 
tools for specific product cases. This work focuses 

on understanding user needs for the design of a lamp 
customization tool for citizens who have growth, 
experience, success, materialism, and enjoyment as core 
values. Our main goal is to define the requirements 

for a customization tool of self-designed lamps by (1) 
exploring user needs and customizable attributes; (2) 
prioritizing and selecting a set of attributes and needs 
to define requirements for a lamp customization tool; 
and (3) proposing a design to cover these requirements. 
User requirements were drawn from four sources. A 
Literature Review (LR) provided general requirements. 
Contextual Inquiries (Cl) explored users’ thoughts and 
interests on customizing products. A participatory study 
(Co-constructing Stories, CCS) gained more insights 
into the customization process of lamps in particular. 
Next, a Survey (S) was conducted to prioritize user 
needs and to define a final set of requirements. Finally, a 
design of a customization tool using these requirements 


is presented. 


2. Literature Review 


Consumers are willing to pay extra for products that 
they have customized according to their preferences 
[1]. The added value of customization can be explained 
with the Ikea effect [2]. However, the effort put into 


the customization process alone does not increase the 


Fig. 1. ^ mockup of 

an Augmented Reality 
(AR) customization 
tool design providing 
different starting 
points, a limited set of 
customizable attributes, 
and increasingly realistic 


visualizations. 


3D model presentation 
in context using AR 


perceived product value. An enjoyable customization 
process and a high preference fit are required as well to 
result in a higher subjective product value when using 
mass-customization toolkits [1]. Product customization 
requires consumers to build their own product. The 
difficulty of decision-making increases as the number 

of alternatives and attributes increases, if a specific 
attribute is difficult to process, or if there is uncertainty 
about the values of attributes [3]. Considering too 
many options can lead to decision-making errors 
because there is too much to decide [4]. An optimal 
limit of options was found to be six when choosing 
among gourmet jams and chocolates [5]. Dysfunctional 
effects of information overload emerge with ten or 
more alternatives when choosing houses [6]. The order 
of attribute presentation is also influential. When 
attributes with relatively few options follow attributes 
with relatively many options people are more likely to 
accept default options in the context of custom-made 
suits and automobile choice [7]. The perceived comfort 
and preference fit increase with the user's expertise in 
the context of consumer laptop computers [8]. Not all 
customers are interested in fully exploiting the potential 
of customization. Hence, several initial designs should 
be provided as starting points [8]. By their very nature, 
customized products are likely to be unique. This makes 
it difficult for consumers to anticipate their post- 
purchase experience [8]. Additionally, several product 
attributes can be intangible which makes it hard to show 
or explain them on a screen. Also, the context in which 
the products are presented is important. Because of 
this, AR systems overall satisfaction is higher than when 


using traditional e-commerce stores [9]. 


Inspiration and help, freedom and 
support, trusworthy visualizations 


Hands-down comfortable 
customization 


3. User Studies 


3.4 Methods 


Participants. 12 volunteers (25-66 yo, 3 designers) 
visiting, buying, or selling products that could be 
customized during the Dutch Design Week participated 
in the Cl. 10 volunteers (5 female, 5 novices) 
participated in the CCS. 29 participants completed the 
Survey (12 male, Mean = 28.29 yo, SD = 2.49). Detailed 


protocols are available in [10]. 


Procedure Cl. We observed attendants and inquired 
about people’s preference to customize products 

and their previous experiences with it. Additionally, 
designers were asked whether they usually set design 
limitations for users in the customization process and 


why. 


Procedure CCS. In the sensitizing part, questions 
about previous experiences choosing and buying lamps 
and ideas about lamp customization were asked. In the 
elaboration task, participants were asked to design 
their own lamp while thinking-aloud. Diverse materials 
and tools were provided to inspire participants and 

to enable the observation of the participants’ use 

or interest in (1) different levels of abstraction; (2) 
prototyping and visualization of tangible and non- 
tangible tools; and (3) possible lamp attributes. 
Participants used both lo-fi and hi-fi prototyping tools 
or a combination of both. For example, participants 
would draw the shape of their lamp on paper and use a 
real fabric to explain the material. Also, the attributes 
or needs mentioned by the participants and the levels 
of abstraction of the tools and their tangibility was 


registered. 
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Procedure Survey. 19 customization aspects elicited 
in the CCS were rated in random order on a 5-point 


Likert scale (5: extremely useful). 


3.2 Results and Discussion 


The contextual inquiry confirmed several customization 
requirements suggested by the literature. Most 
participants value the concept of customization, 

but they still find having a competitive price crucial. 
People valued the customization of both functionality 
and style. Style features should promote uniqueness 
and creativity. Already made products should be also 
available. Visualizations of the products should provide 
accurate feedback to let the user see the impact of 
every choice on the final product. People care about the 
quality of the end product, so they often seek advice 
from experts and they value their personal connection 
with them. "You need skills, if you cannot visualize it, the 
disappointment could be big." - Garment customization 


designer 1 


In the co-constructing stories sessions we observed 
that trustworthy, full and realistic impressions about the 
product should include light effects. Decisions on the 
customization of technical parts of the lamp and other 
functionalities should be made by an expert. Moreover, 
users know what they do not want, but they are not 
certain about what they actually want. Thus, examples 
are useful to imagine the final product. Additionally, 
support in the decision-making process is required, 

by providing a good overview over all the options and 
allowing comparison of different products. Next, users 
should be able to iterate and decide the order they 
want to follow by themselves. They should also choose 
from a starting point: from scratch, from a basic model, 
or from a pre-designed alternative. A summary of the 


obtained requirements is shown in Table 1. 


The survey ratings of each customization variable 
were ranked in order of importance. The top needs 
and attributes were the beauty of the product, the 
type of the lamp, the purpose of the lamp, the type of 
light, the fact that the lamp matches the interior, and 
price. These are mainly need-based attributes. This is 
probably because the respondents were novices [13]. 
Nevertheless, to support expert users as well, these 


need to be translated to parameter-based customization 
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variables. Therefore, we suggest to use lamp shape, 


color, material, and lamp dimensions as parameters 


that can contribute to create a combination that can be 


beautiful for each particular user. 


General requirement Description Source 
Limited options at a time Between 6 and 11 options at a time. LR, CI 
Different starting points (1) scratch or free-form interface; (2) a | LR, CI 
for different users model or combined configuration; and 
(3) a pre-designed alternative. 
Trustworthy visualizations | Increasingly richer, realistic LR, Cl, 
in context visualizations in context to let the user | CCS 
understand the final outcome. 
Overview of the options Side-by-side comparison of saved CCS 
for decision-making configurations and their characteristics. 
support 
Price and value Price visible to account for every CIS 
change. 
Customer-designer A designer or expert should be CCS 
relationship support available to guide or advise the 
customer. 
Post-editing and iteration Iterations in the design process, CCS 


possibility 


bookmarks, and going back to previous 


designs. 


Table 1. Final user requirements. The sources are LR: Literature 


Review, Cl: Contextual Inquiry, CCS: Co-Constructing Stories, S: 


Survey 


4. From Requirements to Design 


The user requirements can be directly translated into 


design elements in an AR customization application. 


First, AR allows to place and show the product in 


context. A 3D model of the lamp can be placed and 


visualized in the desired room position (Fig. 1 (a)). 


The realism of the 3D model can be increased as the 


user selects the desired attributes. However, the 


final visualization should be trustworthy and exactly 


represent the product they are going to receive. 


Second, multiple starting points should be provided 


for different types of users. Users can explore pre- 


designed lamps or start from scratch with their own 


design. Additionally, tips on each customizable attribute 


could be shown to provide support to the user. Users 
can change each attribute in the order they like and 

go back and forth. However, the set of customizable 
attributes should be limited each time. In the example 
shown in Fig. 1 (b), there are only six customizable 
attributes: shape, size, material, color, and light settings. 
After selecting one of them, only few options for that 
attribute are shown around the lamp 3D model. Finally, 
it is important to prevent fatigue when using AR. 
Therefore, users should always be able to switch from 
a Static- to Live-view. The Live-view provides real time 
visualization of the lamp in AR, allowing the user to see 
different lamp perspectives. The Static-view shows a 
picture representing one particular view of the room as 


a background for the lamp customization (Fig. 1 (c)). 


5. Conclusion 


This work defined a set of relevant requirements of 
user needs for an AR lamp customization tool. These 
requirements are (1) inspiration and help; (2) freedom 
and support during the customization process; and (3) 
trustworthy visualizations. From the broad and generic 
guidelines available, the methods applied enabled us to 
specify a set of concrete attributes and needs to meet 
user needs when customizing lamps in context using 
Augmented Reality. Such methods and techniques can 
also be used to define specifications for other types of 


products. 
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Speculating on the Future of Graphic 
Design in the Age of Intelligent Machines 


Abstract 


Machine learning is set to have a profound impact on 
the graphic design industry in the near future. Despite 
its proximity, graphic design education and practice 

are largely sidelined from participating in the highly 
scientized spheres of computational aesthetics and 
applied image processing. Within this context, designer 
Sekyeong Kwon sought to make visible some of the 
cultural and practical implications of Al-powered design, 
from a graphic design perspective. The resulting practice- 
led project, Michael Barnes, falls within the subfield of 
adversarial design, and seeks to provoke contestation 
and debate around automation in design. The following 
short paper briefly sketches out the current graphic 
design landscape in relation to emerging technologies; 
outlines Michael Barnes; and explores a number of issues 
raised by the project including questions around (inter 


alia) aesthetics, authorship, and representation. 


Keywords 


Graphic Design, Automation, Adversarial Design, 


Practice-Led, Authorship 


1. Graphic Design and Automation 


The authorial hand of the graphic designer has always 


been indissolubly linked to the tools and technologies 
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available to them. Since the mid-twentieth century, this 
causal relationship has been increasingly influenced by 
developments in human-computer interaction (HCI), and 
more specifically, ‘graphical user interfaces’ [1]. MIT's 
Computer-Aided Design Project (1959-1967), for example, 
which spearheaded much of the early work into these 
systems, sought to create an interface which would 
"couple a man and a machine into a problem-solving 
team for fresh design problems" where each would 


perform "better than ... man or machine alone" [2]. 


More recently, however, and with developments in 
machine learning and deep learning in relation to 
computational aesthetics and aesthetic computing (see, 
for example, Google's DeepDream, MIT's Nightmare 
Machine) the view of computer-aided design software 

as an aid, is being superseded by its potential to make 
autonomous or semi-autonomous creative decisions [3- 
4]. Applied instances of embedded artificial intelligence 
(Al) technology within the field of graphic design 
currently include Adobe's Creative Cloud software, 
which is able to analyze the content of an image or video 
and make "intelligent recommendations" in order to 
automate "time-consuming" aspects of design [5]. And, 
looking ahead, Autodesk is presently developing Project 
DreamCatcher — a generative design system that will be 
capable of producing thousands of design options in a 
matter of seconds and "play an active, participatory role 


in the invention of form" [6]. 


Fig. 1. Michael Barnes 
Work (Source: https:// 


barnes.persona.co/) 


Fig. 2. Michael Barnes 
Biography (Source: 
https://barnes.persona. 
co/) 


Of course, these advancements have elicited polarizing 
views in the field of design. Typically, debates center 
around employability and economic productivity, 

with factions on one side arguing that by relegating 
time-consuming production activities to computers, 
designers will be able to expedite their creative work 
[7]. Others, however, warn of a dystopian scenario with 
widespread unemployment resulting from increased 


automation [8]. 


However, outside of this binary rhetoric, and the pitting 
of the ‘technophobic humanist’ against the ‘inhuman 
technologist’ [9], what is less often discussed are the 


potential ramifications of Al technology in relation to 


the craft of graphic design practice. 


SELECTED 
WORKS 


The Vanishing Future 
Visual Identity 


A ‘The Vanishing Future’ is a design festival in Seoul that needed a new 
logotype. It consists of a poetic visual identity that is both meaningful and 
classic. Main image is used as an advertising, a symbol to develop the 
functional ethos of festival. 


MICHAEL 
BARNES 


Michael Barnes is an award-winning design critic and visual communicator 
based in San Francisco and New York. His work places an experimental 
emphasis on installation, digital art, web design and typography. Ever since 
he moved to London, he started to seek finding ways to apply emerging 
technologies to encourage our humanness, and ultimately to design a better 
world. He holds an MA in Communication Design from Falmouth university. 


Within this rapidly-evolving and highly-contested 
environment, designer Sekyeong Kwon sought to 
encourage debate and make visible core concerns 
around Al-powered design, from a graphic design 


perspective. 


The resulting practice-led project, Michael Barnes, is 


outlined below. 


2. Michael Barnes 


Michael Barnes is a self-titled portfolio website (https:// 
barnes.persona.co/) which includes a manifesto, a 
curriculum vitae, and a gallery of design work featuring 


corporate identity, packaging and branding projects (See 


= futureismichael@gmail.com 
© +44 (0)7380 262154 


MANIFESTO - WORK - BCV 


= futureismichael@gmail.com 
© +44 (0)7380 262154 


MANIFESTO - WORK - &CV 
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Fig. 1). According to his biography, Barnes' is "an award- 
winning design critic and visual communicator based in 


San Francisco and New York" (See Fig. 2). 


Despite the unremarkable design work and the 
uncomfortable syntax, the website seems to all intents 
and purposes authentic. That is, the portfolio appears 
typical of the countless online design portfolios that are 
(arguably) largely indistinguishable from one another 
[10]. 


However, 'Michael Barnes', his name, biography, 
manifesto, and projects are entirely fictitious. 'He' has 
been computationally generated using a combination of 
tools (Python coding, Markov Chain sequencing, image 
generators, etc.) and data scraped from existing graphic 


design texts and websites. 


Thus, 'Barnes' represents but one randomized version 
of a potentially exponential number of designers 

and portfolios that could have been algorithmically 
generated. He could have, for example, been just as 


easily characterized as: 


... a curious interaction designer and UI/UX designer 


based in Barcelona and Bangkok... 


Or: 


... a meticulous design strategist and visual 


communicator based in London and Sydney... 


Certainly, this kind of parafictional deceit in cultural 
production is nothing new; Nat Tate (created by William 
Boyd) and The Yes Men (Jacques Servin and Igor Vamos) 
spring to mind. In each case, the characters draw on and 
mimic existing dialects — cultural, aesthetic, and textual 
— as a means of gaining entry to, and legitimacy within, 

a specific sphere [11]. However, unlike The Yes Men (et 
al) who intentionally ‘dupe’ the viewer, ‘Michael Barnes’ 
offers up 'his' own duplicity for scrutiny. Namely, the 
last section of the website, the curriculum vitae, reveals 
the project as a fictional endeavor, and makes publicly 
available the various code and tools used in the creation 


of the character and his portfolio. 
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3. An Adversarial Approach 


While not overtly political, Kwon's project can be 
seen as a form of adversarial design — a type of critical 
making which seeks to provoke debate through the 
speculative modelling of possible scenarios and socio- 
political configurations [12]. Specifically, once 'Barnes' 
is revealed as an algorithmically generated figure, the 
existing relationship between Al technology and design, 
and perhaps more importantly, its shared future, are 
visibly problematized for the viewer. By doing so, 
rather than simply exploring the potential applications 
of Al technology and design, Michael Barnes demands 


consideration of its implications as well. 


One area highlighted by Michael Barnes is the ease with 
which it achieves, at least in part, a kind of semiotic 
invisibility. It appears authentic. Thus, the website 
exposes the generic globalized reality of design and 
portfolio websites in which imitation has become a tool 
of legitimization. This, in turn, raises further questions 
around the current impact of Al technology in the field 
of design. Firstly, given that the featured work (‘best of’, 
'most viewed' etc.) on graphic design showcase sites 
(for example, Behance and Dribbble), and the practice 
of locating stock images and templates, is search engine 
driven, to what degree are computers already agents of 
(rather than simply aids to) design practice? And looking 
forward, if design software is increasingly left to make 
decisions computationally, what will these be based on? 
Aggregates of taste? Engagement levels? Sponsorship? 


And, in turn, whose views will these privilege? 


As such, the project recognizes that advancements 
in graphic design Al are not value-free, but rather, 
embedded within a broader data environment, which 
will ultimately prioritize particular social, political and 


economic forces. 


Another area underscored by the project are shifting 
notions of originality within our current hyper- 
networked culture. Specifically, while Michael Barnes' 
portfolio is comprised entirely of pre-existing data, it 

is original in the sense that it is one-of-a-kind. In turn, 
this begs the question: who is the author of the website? 
The algorithm? Kwon? The ‘original’ authors of the 
now-unrecognizable data sources? And, additionally, 


who owns the intellectual property rights to the work? 


While theorization around shared cultural production 
and ownership recognizes contemporary appropriative 
cultural practices and the use of existing data as 
'material' for production (such as sampling by deejays), 
the discourse is still predicated on human-agency 

and intentionality, which does little to clarify ongoing 


debates around Al and algorithmic authorship [13-14]. 


4. Contestation vs Consensus 


Michael Barnes is by no means a ‘polished’ form of 

Al; the design and various textual elements are 

largely inexpert; and the final outcome relied on the 
assembly of its individual parts by Kwon. However, the 
intention of the project is not to predict the future, 
nor to attempt to compete with rapid technological 
advancements in the field of computer science. Rather 
it seeks, playfully, to provoke debate and speculate 
‘what if?'. By suggesting a possible world, where a new 
‘designer’ complete with personal backstory, ethos and 
portfolio can be generated at the click of a button, ad 
infinitum, the project encourages a recognition of the 
tensions which lie at the heart of the convergence of Al 


and graphic design. 


In turn, Michael Barnes seeks a response from design 
educators and practitioners, a community whose 
voice is more often than not lost in the scientized race 
for Al. This provocation is, perhaps, most succinctly 
encapsulated in the darkly comic automated email 


response should you try to get in touch with ‘him’: 


Hello! Thank you for your email. 


BAD NEWS: I'm away from the office until Artificial 


General Intelligence becomes a reality. 


GOOD NEWS: Me being away means that your job is 


safe from automation — for now. 


Until then, Cheers! 
Michael 
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Al-Stylist: An Al-based Framework for 
Clothing Aesthetic Understanding 


Abstract 


Current studies in computer science on clothing 
aesthetic evaluation are difficult to be practically applied 
as fashion is a subjective field affected by various 
factors, like cultural background, or physical information 
(e.g. skin color). On the other hand, computer is 

good at processing logical tasks with clear definition 

and evaluation index. In this work, we propose a 
framework which divides the aesthetic evaluation task 
into three levels, namely apparel harmony, personal 
compatibility, and personality expression, to reduce 

its complexity such that computer has great potential 

to evaluate the clothing aesthetic practically. Each 

level is clearly defined, and the technic solutions is 
proposed correspondingly. Initial work with examples is 
presented to demonstrate the potential of the proposed 


framework. 
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1. Introduction 


The cross-domain between fashion and Artificial 


Intelligence (Al) has received increasingly attention 
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recently. In the field of computer science, fashion 
related tasks can be roughly divided into three 
progressive parts [1]: fine-grained attributes recognition 
for fashion lovers [2-4], clothing aesthetic evaluation for 
stylists [5-7], and design auto-generation for designers 
[8-10]. With the assistance of Al, huge amount of 
personal information can be recorded and processed 
simultaneously. It can be expected that customized 
advices of outfit matching or personal styling service 
can be easily accessible by the public. However, the 
challenge is to what extent an intelligent model can 
behave like a human stylist with professional styling 


knowledge. 


Fashion is an extremely subjective topic and highly 
diversified among different individuals. Current 
evaluation studies also suffer from this subjective 
factor and thus cannot achieve good performance in 
real application. In this work, we attempt to abstract 
the subjective aesthetic evaluation and divide this 
complicated process into a series of small work with 
clear task definition. We propose a computer based 
clothing aesthetic evaluation consisting of three 
progressive levels, namely apparel harmony, personal 
compatibility, and personality expression with technic 
solutions. An initial work is described to elaborate the 
whole idea of the proposed framework which is like an 
Al-stylist. 


Fig. 1. Proposed 
framework of the Al- 
based clothing aesthetic 


evaluation 


2. Methodology 


In the proposed framework, the three levels are 


Apparel harmony: Are all fashion items of an outfit well 
matched? 

Personal compatibility: Is an outfit matching with an 
individual's physical information e.g. hairstyle, body 


figure, etc.; 


Personality expression: Is the total look well expressing an 


individual's attitude or emotion? 


In computer science, compatibility of fashion items 
(including apparel, shoes, and bags, etc.) of an outfit 
is considered under a matching principle at apparel 
harmony level. The requirement of an Al model at this 
level is to assess the visual balance of an outfit (Good, 
Normal, or Bad) and the model is also capable of 
providing a reasonable explanation for its judgement. 
On the first level, physical information including body 
figure, skin tone, and hair style etc. will be taken into 
consideration. Then, at the second level, i.e. personal 
compatibility level, the Al model is required to know 
how to create an outfit for an individual. Finally, at 
the highest level, i.e. personality expression level, the 
Al model is expected to be able to understand an 
individual's personality and recommend suitable items 


accordingly. 


To achieve the objectives stated above, the proposed 
evaluation model consists of three phases. Specifically, 
the first phase is attribute recognition which provides 
basic fashion knowledge (V-neck, velvet, and dotted 
etc.) to the model. The second phase is rule learning 
which teaches the model about the basic rules of mix 
and match. The final phase is auto introspection to 
further refine its performance as a stylist. The details of 


the proposed framework are shown in Figure 1: 


(1) Phase 1 — clothing attribute recognition and concept 
learning 

Attributes are treated as components of an apparel 
item. High accuracy of fine-grained attribute recognition 
is the foundation for clothing aesthetic understanding. 
Although previous networks reported in the literature 
have achieved promising performance, there is still a lot 
of room for improvement from the fashion perspective. 
There are two main tasks in this phase: 1. Fine-grained 
attribute recognition, especially the accuracy of multi- 
label recognition; 2. Attribute feature presentation to 


connect the visual feature with fashion semantic. 


(2) Phase 2 — experts embedded and reinforcement 
learning 

Based on the recognized clothing attributes, the 
reasoning network based on the recognized attribute 
is proposed. In this phase, the core task is to find the 


relation among attributes of apparel items of an outfit. 


patch work "ed 


Color {yellow, red, orange, 


Color should no more than 


Generate fashion images based 


Fine-grained attributes recognition 


Instances recognition and concept learning 


puff sleeves etc.} 2/3 on the standard. 

Stripe Pattern {stripe, patch work, Pattern should not be The generated examples are 

yellow. off shoulder etc.} complicated used as training data to further 

V-neck Pink chiffon Characteristic {V-neck, lapel, Balmain sleeves should not adjust the standard based on 

orange lapel etc.} exist with peter pan collar feedback from the discriminator. 
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concept learning sense correction standard : 


ME 


In terms of mix and match, color is 


For example, this is red and red is the most effective attribute 
a color dimension 

What attributes has more effect on Color > pattern > material = 
mix and match characteristic 


Rule embedded reasoning network 


Judge and comment of the outfit 
to verify and correct the standard 
Converting the standard to the 
aesthetic taste by the practical 
application 


Practical verification with GAN 


School(supervised): attributes recognition and rule learning 
Causal Inference 


Society(unsupervised): auto introspection 
Probability Statistic 
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Current approaches are mainly focusing on finding a 
compatibility space and suitable metric to measure 
the distance between attributes. We propose a novel 
method with symbolic reasoning to formulate this 


problem. 


(3) Phase 3 — practical verification with Generative 
Adversarial Nets 

As shown in Figure 1, both attribute recognition parts 
and rules learning parts are under supervised learning 
or semi-supervised learning which requires training 
data. (It is noted that the rule defined here is a kind of 
principle for guidance only and thus is not the same as 
the rules in the previous expert system.) To fine-tune 


the performance of evaluation model and generate 


Bad. The outfit looks 
messy with too much 


Bad. Light blue does not match 
with medium green as both colors 


more data for training, a GAN-based model can be 


developed to realize the auto introspections. 


3. Initial Experiment 


To demonstrate the potential of proposed framework, 
we present some experimental results of the initial 
work on clothing aesthetic evaluation at the apparel 
harmony level (first level of the framework) for the 
reference. Different from the scenario like Amazon 
echo look (which is limited to make a comparison on 
two outfits), the proposed framework should provide 
absolute evaluation/assessment on one outfit rather 
than comparing two outfits in most real-life situation, 


e.g. someone always asks how is my outfit today? As 


Good. Floral print gives soft 
and feminine feelings. Black 
color in the bottom acts as a 


Fig. 2. Three ranks of 


outfit aesthetic ranking 
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shown in Figure 2, three ranks are defined based on 
fashion and styling knowledge to evaluate an outfit. 
The dataset built for training is based on these three 
defined ranks. We collected the image source from 
Polyvore firstly. Then, the collected outfits were 
annotated with the three-evaluation ranks (Good, 
Normal, Bad) and its corresponding reasons. We show 


the samples of the proposed dataset in Figure 3. 


Next, we implemented a rule expert as a decision 

tree program. The learning system contains two 
components: a recognition network and a rule expert 
(The structure of the rule expert is shown in Figure 
4). The recognition network fits into a traditional 
3-class classification task. The evaluation loss is to 
train the network supervised by the labels. To let the 
network generate a reason why the outfits is rated as 
Good, Normal or Bad, we transform the knowledge of 
providing a reason from the rule expert to the network 
by multi-task learning. Finally, evaluating/grading an 
outfit with absolute judgement in terms of reasonable 
explanation can be generated. Sample results and 


compared approaches can be seen in Figure 5. 


Fig. 5. Examples of the initial result 
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4. Conclusion 


Recently, more and more researchers from computer 
science society put their efforts on applying Al 
technologies to fashion related tasks. However, 
fashion and styling are a very subjective topic and thus 
conventional computational techniques cannot be 
directly and practically applied to evaluate the clothing 
aesthetic. In this paper, we propose a framework at 
three progressive levels, including apparel harmony, 
personal compatibility, and personality expression 

to address this issue. Initial experimental results at 
the harmony level are presented to demonstrate the 
potential of the proposed framework. The proposed 
framework serves as a foundation and guidelines for 
the future development of related aesthetic evaluation 


techniques and systems which can be practically applied. 
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